New to the community? Start here

Post Indexer not indexing. Is cron jobs needed?

Hi,

Post Indexer version: 3.0.6.1

8 Sites in queue, and it has been like that for multiple days. I read something about Post Indexer requires cron jobs active to work. Is that true? It is not possible to somehow manually get the plugin to index?

[attachments are only viewable by logged-in members]

Wed, 15 Jul 2015 11:59:15
  • Predrag Dubajic
    • Support

    Hey @Karsten,

    Hope you’re doing well today :slight_smile:

    Can you make sure the post is published on a site with privacy set to “Allow search engines to index this site.” (Found in the regular admin area — not the Network Admin area — under Settings > Reading.)

    If that doesn’t give any results would you mind allowing support access so we can have a closer look at this?

    To enable support access you can follow this guide here:

    https://wpmudev.com/manuals/wpmu-dev-dashboard-enabling-staff-login/

    Best regards,

    Predrag?

  • Karsten
    • Flash Drive

    Hi Predrag,

    I'm fine, thanks. Thanks for the quick reply.

    I have checked the settings and search engines are allowed on all sites.

    Support staff access is now successfully enabled!

    [attachments are only viewable by logged-in members]

  • Predrag Dubajic
    • Support

    Hi @Karsten,

    Thanks for granting access, however there is still authorization required to access your page, could you please disable this while we debug this or send me your login details via email so we can check it out.

    You can send us your details using our contact form https://wpmudev.com/contact/ and the template below:

    IMPORTANT: Make sure you select “I have a different question” for your topic so it doesn’t go back to forums – this and the subject line ensure that it gets assigned to me.

    Subject: “Attn: Predrag Dubajic“

    – WordPress admin username

    – WordPress admin password

    – login url

    – Authorization Required popup username

    – Authorization Required popup password

    – link back to this thread for reference

    – any other relevant urls

    Looking forward to your response.

    Best regards,

    Predrag

  • Karsten
    • Flash Drive

    I send the information ealier today.

    I did some more testing, and when I removed the .htaccess protection, it started indexing.

    Is it the case that Post Indexer does not work when protection your site with e.g. .htaccess? And if yes, is there a workaround?

  • Adam Czajczyk
    • Support Gorilla

    Hey Karsten,

    I hope you’re well today!

    As Post Indexer works “kind-a-like” a regular web spider/crawler the .htaccess protection may indeed prevent it from accessing the site which results in sites not being indexed. I took a look at plugins code and it turns out that although most of the “indexing queries” are performed directly against database (on a WP core level) it’s also necessary to access feeds produced by your network’s sub-sites.

    If these sub-sites (and that means also feeds) are protected, they won’t be accessible for a plugin and won’t be indexed. The only way around I can think of at the moment would be to allow Post Indexer to authorize itself but this would require a serious rebuilding of a plugin code so that would be more like a custom job.

    That being said, if you’d protect only a back-end of your site’s with .htaccess it should help. You could also try to exclude addresses of the feeds from protection, however I’m not quite sure about it. It might be worth giving a try though :slight_smile:

    Cheers,

    Adam

  • Karsten
    • Flash Drive

    Hi Admin,

    Thanks for the reply!

    I tried your last suggestion allowing access to the different feeds. I did it using this function in the .htaccess file:

    SetEnvIf Request_URI "(feed/)$" allow
    
SetEnvIf Request_URI "(feed)$"  allow
    
SetEnvIf Request_URI "(feed/rss/)$"  allow
    
SetEnvIf Request_URI "(feed/rss)$"  allow
    
SetEnvIf Request_URI "(feed/rss2/)$"  allow
    
SetEnvIf Request_URI "(feed/rss2)$"  allow
    
SetEnvIf Request_URI "(feed/rdf/)$"  allow
    
SetEnvIf Request_URI "(feed/rdf)$"  allow
    
SetEnvIf Request_URI "(feed/atom/)$"  allow
    
SetEnvIf Request_URI "(feed/atom)$"  allow

    I also added the same lines for all subsites. I’m still having the same issue, so I guess the next step would be to have someone modify the plugin.

  • Karsten
    • Flash Drive

    Hi Predrag,

    This is the entire content:

    AuthType Basic
    
AuthName "Auth Required"
    
AuthUserFile /customers/f/5/c/crisp.one/httpd.www/.htpasswd
    
Require valid-user
    
# Allow public access to the following resources
    
# Dashboard
    
SetEnvIf Request_URI "(/feed/)$"  allow
    
SetEnvIf Request_URI "(/feed/rss/)$"  allow
    
SetEnvIf Request_URI "(/feed/rss2/)$"  allow
    
SetEnvIf Request_URI "(/feed/rdf/)$"  allow
    
SetEnvIf Request_URI "(/feed/atom/)$"  allow
    
SetEnvIf Request_URI "(/feed)$"  allow
    
SetEnvIf Request_URI "(/feed/rss)$"  allow
    
SetEnvIf Request_URI "(/feed/rss2)$"  allow
    
SetEnvIf Request_URI "(/feed/rdf)$"  allow
    
SetEnvIf Request_URI "(/feed/atom)$"  allow
    
# Plugins
    
SetEnvIf Request_URI "(/wp-content/plugins/)$" allow
    
SetEnvIf Request_URI "(/wp-content/plugins)$" allow
    
Order allow,deny
    
Allow from env=allow
    
Satisfy any
    

    
RewriteEngine On
    
RewriteBase /
    
RewriteRule ^index.php$ - [L]
    

    
# Add a trailing slash to /wp-admin
    
RewriteRule ^([_0-9a-zA-Z-]+/)?wp-admin$ $1wp-admin/ [R=301,L]
    

    
RewriteCond %{REQUEST_FILENAME} -f [OR]
    
RewriteCond %{REQUEST_FILENAME} -d
    
RewriteRule ^ - [L]
    
RewriteRule ^([_0-9a-zA-Z-]+/)?(wp-(content|admin|includes).*) $2 [L]
    
RewriteRule ^([_0-9a-zA-Z-]+/)?(.*.php)$ $2 [L]
    
RewriteRule . index.php [L]

    Thank you :slight_smile:

  • Jose
    • Bruno Diaz

    Hello there Karsten,

    Hope you are doing great today,

    WP Cron is based on a loopback request to yourdomain.com/wp-cron.php.

    In order for the cron to be executed, you need to exclude the file wp-cron.php from the protected files.

    Please try adding this into your .htaccess:

    <Files "wp-cron.php">
    
Allow from All
    
Satisfy Any
    
</Files>

    Let me know if this solves the problem.

    Cheers,

    José