The ASUS RT-AC87U supports multicast and BT's TV service and is an absolute doddle to set up. If you are using standard ASUS firmware, then select 'LAN' in the 'Advanced Settings'. Then select the 'IPTV' tab near the top of the screen. Simply use the settings shown in the image below, with 'Microsoft' for 'Use DHCP routes' and 'Enable multicast routing (IGMP Proxy)' and 'Enable efficient multicast forwarding (IGMP Snooping)' enabled and the number zero for 'UDP Proxy (Udpxy)' . That's it! Multicast should work perfectly, but bear in mind that it will only work via Ethernet and not WiFi.
How to check it works
If your BT set-top box is installed, you can simply reboot the box and you should be able to access the multicast channels via the Electronic Programme Guide.
If you have a PC or Mac connected to your Asus RT-87U router using Ethernet, you can use a media streaming application such as VLC to view BT's test channel. In the VLC programme, select 'File' and then 'Open Network...', and then 'Open RTP/UDP Stream'. Then in the options select 'RTP' and 'Multicast' with IP address 126.96.36.199 and port number 5802. A window should pop up showing the multicast stream! Please note that your PC or Mac needs to be connected to your Asus router via Ethernet cable as multicast channels generally do not work via a WiFi connection.
If you have a Billion 8800NL or 8800AXL, you may be interested in viewing our Setting Up the Billion 8800NL and 8800AXL For BT TV (Multicast) guide.