1

I am making an inventory management site using Angular and Firebase. Because this is angular, there are problems with web crawlers, specifically Slack/Twitter/Facebook/.etc crawlers that grab meta information to display a card/tile. Angular does not do well with this.

I have a site at https://domain.io (just the example) and, because of the angular issue, I have a firebase function that created a new site that I can redirect traffic to. When it gets the request (onRequest), I can grab whatever query parameters I've sent it and call the DB to render the page, server-side.

So, The three examples that I need to redirect are:

From: https://domain.io/item/ABC123
To:   https://us-central1-domain.cloudfunctions.net/metaTags?item=ABC123

--

From: https://domain.io/bench/USERNAME
To:   https://us-central1-domain.cloudfunctions.net/metaTags?bench=USERNAME

--

From: https://domain.io/bench/USERNAME/ITEMTYPE
To:   https://us-central1-domain.cloudfunctions.net/metaTags?bench=USERNAME&type=ITEMTYPE

I can't seem to get the right combination. This is what I have right now. Note: The item redirect is working, but the other two are not...

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteRule ^index\.html$ - [L]

  #  If a bot goes to an Item
  RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|Slackbot|Slack-ImgProxy|Slackbot-LinkExpanding|Site\ Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram [NC,OR]
  RewriteCond %{REQUEST_URI} ^/item/
  RewriteRule ^item/(.+)$ https://us-central1-domain.cloudfunctions.net/metaTags?item=$1 [NC,L]

  #  If a bot goes to a simple bench
  RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|Slackbot|Slack-ImgProxy|Slackbot-LinkExpanding|Site\ Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram [NC,OR]
  RewriteCond %{REQUEST_URI} ^/bench/
  RewriteRule ^bench/(.+)$ https://us-central1-domain.cloudfunctions.net/metaTags?user=$1 [NC]

  #  If a bot goes to a bench of a type
  RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|Slackbot|Slack-ImgProxy|Slackbot-LinkExpanding|Site\ Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram [NC,OR]
  RewriteCond %{REQUEST_URI} ^/bench/
  RewriteRule ^bench/(.+)/(.+)$ https://us-central1-domain.cloudfunctions.net/metaTags?user=$1&type=$2 [NC]

  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteRule . /index.html [L]
</IfModule>

It's also important to note that I would like this to work on any sub-domain (I have my dev and staging environments set up with subdomains) as well as make sure it's always directed to https

Any thoughts?

2
  • Can you look at the issue from another perspective? Angular doesn't do well with web crawlers because the rendering happens in the browser and crawlers usually just parse the generated html, which will not contain much when viewed in a regular angular application. But if you use Angular Universal to render pages server-side, your angular app will be crawler-friendly without the need to think about redirections. Commented Dec 2, 2019 at 20:40
  • @DejanJanjušević, Thank you for the reply, though I spent 4 days trying to set up Angular Universal, only to realize that the server I need to run on is an Apache server where I don't have any control over Node resources (or adding firebase PHP functionality to it). That's why I went with Firebase's onResource functionality, because it hosts it's own page that would act like a universal SSR page. If I go to the link directly, it does exactly what I want, but because of the environment I need to be on, I just need to write the correct htaccess file to redirect specific bots
    – ntgCleaner
    Commented Dec 2, 2019 at 20:43

2 Answers 2

1
+100
  1. Use [NC,L] flags also for both bench RewriteRules
  2. Use ([^/]+) instead of (.+) in regex patterns
  3. Change [NC,OR] to [NC] in user-agent RewriteCond
8
  • So, this is redirecting to the correct spots, but unfortunately, it redirects me (as a human, not a bot) as well
    – ntgCleaner
    Commented Dec 2, 2019 at 21:36
  • Change [NC,OR] to [NC] in user-agent RewriteCond Commented Dec 2, 2019 at 21:38
  • Makes me want to understand htaccess more! Such simple solutions to this stupid problem! Thank you for your help. I am allowed to award bounty in 22hrs. Would you mind adding your last comment to your answer as "step 3"? I will award this when I am able - THANK YOU
    – ntgCleaner
    Commented Dec 2, 2019 at 21:43
  • So, I have this implemented now and it's working for the crawlers, but it's not redirecting HTTP -> HTTPS anymore. Would there be something I'm missing here?
    – ntgCleaner
    Commented Dec 16, 2019 at 19:12
  • There is nothing in above rules related to HTTP -> HTTPS redirection, at least not in the part of .htaccess that you showed us. Perhaps you can edit the question and paste full .htaccess? Commented Dec 16, 2019 at 22:31
0

You can try this below code. (Not tested)

#  If a bot goes to a simple bench
  RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|Slackbot|Slack-ImgProxy|Slackbot-LinkExpanding|Site\ Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram [NC,OR]
  RewriteCond %{REQUEST_URI} ^/bench/(.*)
  RewriteRule ^bench/(.*)$ https://us-central1-domain.cloudfunctions.net/metaTags?user=$1 [NC]

  #  If a bot goes to a bench of a type
  RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|Slackbot|Slack-ImgProxy|Slackbot-LinkExpanding|Site\ Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram [NC,OR]
  RewriteCond %{REQUEST_URI} ^/bench/(.*)/(.*)
  RewriteRule ^bench/(.*)/(.*)$ https://us-central1-domain.cloudfunctions.net/metaTags?user=$1&type=$2 [NC]

If it doesn't work, try [NC] with [L].

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.