IPsec Firewall

If there is no NAT involved, you need:

UDP port 500 for the IKE control channel
IP protocol ESP (50) for the data channel.

   <rule-set name="IPsec" source-interface="pppoe" target-interface="self" no-match-action="continue" comment="Non-NATed IPsec connections from PPP to the Brick">
      <rule name="IKE" target-port="500" protocol="17" action="accept" comment="Internet Key Exchange"/>
      <rule name="ESP" protocol="50" action="accept" comment="Encapsulating Security Payload - encryption protocol"/>
   </rule-set>

However, more likely if your devices are out on the road:

If NAT has been detected, or you force IKE to believe NAT is present (see below) you need:

UDP port 4500 (IKE)
(no need for protocol ESP).
You may need UDP port 500 too, to allow the initial contact to be made - though as soon as NAT is detected (or it has been forced) IKE switches to 4500.

The config force-NAT option [Note the capitalisation ] can be used to force IKE to treat the connection as if it was NATed. This is in the top-level ipsec-ike config item and you need "Show all" on the UI.

Here is an example rule set for allowing IPsec in to a FireBrick:

   <rule-set name="IPsec" source-interface="pppoe" target-interface="self" no-match-action="continue" comment="Allow NATed IPsec connections from PPP to the Brick">
      <rule name="IKE" target-port="500 4500" protocol="17" action="accept" comment="Internet Key Exchange"/>
   </rule-set>

You can join the two rules to create a set that will work for NAT and NON-NAT:

   <rule-set name="IPsec" source-interface="pppoe" target-interface="self" no-match-action="continue" comment="Allow NATed and Non-NATed IPsec connections from PPP to the Brick">
      <rule name="IKE" target-port="500 4500" protocol="17" action="accept" comment="Internet Key Exchange"/>
      <rule name="ESP" protocol="50" action="accept" comment="Encapsulating Security Payload - encryption protocol"/>
   </rule-set>