[Addon] Sitemap

[evilmoe2]

Hello,

I would like to share my sitemap script with you.
My script creates for everey of you categorie a own sitemap xml + a unsorted for everey image without a category.

This script just need 2 files (1 route + 1 template).

I`m using only external storage, you may have to do few changes to this script (from line 82).

Route (upload to app/routes/route.createSitemap.php)

<?php
$route = function($handler) {
    try {
    //var_dump($handler);
  
    $sitemapsFolder = "sitemaps/";
    $absSitemapsFolder = getcwd()."/".$sitemapsFolder;
    $last_pictures   = 50000; // 50k is maximum  https://support.google.com/webmasters/answer/183668?hl=en
    $handler::$vars['sitemapsFolder'] = $sitemapsFolder;
    /////////////////////////////////////////////////////////////////////
    // Basic things
    if( !file_exists($absSitemapsFolder)){
      if(!mkdir($absSitemapsFolder))
        throw new Exception("Can not create ".$sitemapsFolder);
      
       if(!is_readable($absSitemapsFolder) or !is_writeable($absSitemapsFolder))
         throw new Exception("Can not read/write to ".$sitemapsFolder. ". Check permissions.");
    }
    // lastmod
    $objDateTime  =  new DateTime('NOW');
    $dateTime     = $objDateTime->format(DATE_W3C);

    /////////////////////////////////////////////////////////////////////
    // Get categories
    $categories = CHV\DB::queryFetchAll("SELECT category_id, category_name, category_url_key FROM chv_categories");

    // Add unsorted
    $categories[] = array("category_id" => null, "category_name" => "unsorted", "category_url_key" => "unsorted");
    $handler::$vars['categories'] = $categories;

    /////////////////////////////////////////////////////////////////////
    // Create index sitemap on root dir
    $xml = new SimpleXMLElement('<?xml version="1.0" encoding="UTF-8"?><sitemapindex />');
    $xml->addAttribute('xmlns', 'http://www.sitemaps.org/schemas/sitemap/0.9');
    $xml->addAttribute('xmlns:image', 'http://www.google.com/schemas/sitemap-image/1.1', 'http://www.w3.org/2001/XMLSchema-instance');


    foreach($categories as $child) {
      $c = $xml->addChild("sitemap");
  
      $xml_url    = $handler->base_url.$sitemapsFolder.$child["category_url_key"].".xml";
      $xml_local  = $sitemapsFolder.$child["category_url_key"].".xml";
  
      $c->addChild("loc", $xml_url);
      $c->addChild("lastmod", $dateTime );
  
  
      /////////////////////////////////////////////////////////////////////
      // Lets create a sitemap for every category
      $xml2 = new SimpleXMLElement('<?xml version="1.0" encoding="UTF-8"?><urlset />');
      $xml2->addAttribute('xmlns', 'http://www.sitemaps.org/schemas/sitemap/0.9');
      $xml2->addAttribute('xmlns:image', 'http://www.google.com/schemas/sitemap-image/1.1', 'http://www.w3.org/2001/XMLSchema-instance');
  
      $DB = CHV\DB::getInstance();
  
      if($child["category_name"] == "unsorted"){
    
        $q = $DB->query("SELECT image_id, image_title, image_nsfw, DATE_FORMAT(image_date_gmt, '%Y') as year, DATE_FORMAT(image_date_gmt, '%m') as month, DATE_FORMAT(image_date_gmt, '%d') as day, image_storage_mode, image_date_gmt, image_name, image_extension, storage_url FROM chv_images
                          LEFT JOIN chv_storages on storage_id = image_storage_id
                          WHERE image_category_id IS NULL
                          ORDER BY image_date_gmt DESC LIMIT :limit");
    
      } else {
    
        $q = $DB->query("SELECT image_id, image_title, image_nsfw, image_album_id, DATE_FORMAT(image_date_gmt, '%Y') as year, DATE_FORMAT(image_date_gmt, '%m') as month, DATE_FORMAT(image_date_gmt, '%d') as day, image_storage_mode, image_date_gmt, image_name, image_extension, storage_url FROM chv_images
                          LEFT JOIN chv_storages on storage_id = image_storage_id
                          WHERE image_category_id = :category
                          ORDER BY image_date_gmt DESC LIMIT :limit");
  
        $DB->bind(":category", $child["category_id"]);
      }
  
      $DB->bind(":limit", $last_pictures);
      $images = $DB->fetchAll();


      foreach($images as $image){
        //var_dump($image);
        $folder = "";
    
        switch ($image["image_storage_mode"]){
          case "datefolder":
            $folder = $image["year"]."/".$image["month"]."/".$image["day"]."/";
          break;
      
          default:
            $folder = "";
          break;
        }
        // Every image get his own URL
        $cc = $xml2->addChild("url");
        $cc->addChild("loc", $handler->base_url."image/".CHV\encodeID($image["image_id"], "encode") );

        $cc_image = $cc->addChild("xmlns:image:image");
        $cc_image->addChild("xmlns:image:title", $image["image_title"]);
        if($image["image_nsfw"] == 0){
          $cc_image->addChild("xmlns:image:family_friendly", "yes");
        } else {
          $cc_image->addChild("xmlns:image:family_friendly", "no");
        }
        $cc_image->addChild("xmlns:image:loc", $image["storage_url"].$folder.$image["image_name"].".".$image["image_extension"]);
      }
  
  
      $dom = new DomDocument();
      $dom->loadXML($xml2->asXML());
      $dom->formatOutput = true;

      file_put_contents( $xml_local, $dom->saveXML(), LOCK_EX);
      //
    }

    $dom = new DomDocument();
    $dom->loadXML($xml->asXML());
    $dom->formatOutput = true;


    if(file_put_contents($sitemapsFolder."sitemap.xml", $dom->saveXML(), LOCK_EX) === false){
      die("Could not write sitemap.xml");
    }



    } catch(Exception $e) {    
        G\exception_to_error($e);
    }
};

?>

Template (upload to app/themes/Peafowl/views/createSitemap.php)

<?php if(!defined('access') or !access) die('This file cannot be directly accessed.'); ?>
<?php G\Render\include_theme_header(); ?>

<div class="content-width">
    <div class="page-not-found">
        <h1>Sitemap</h1>
        <h4><a href="<?php echo G\get_base_url()."/".get_sitemapsFolder()."sitemap.xml";?>">Sitemap index</a></h4>
    
        <p>Sitemaps created for:</p>
      <div>
        <ul>
        <?php
          $cat = get_categories();
          foreach($cat as $id => $key){
            echo "<li>".$key["category_name"]." = <a href=\"".G\get_base_url()."/".get_sitemapsFolder().$key["category_name"].".xml"."\">".$key["category_name"].".xml</a></li>";
          }
        ?>
        </ul>
      </div>
  </div>
</div>

<?php G\Render\include_theme_footer(); ?>

<?php if(isset($_REQUEST["deleted"])) { ?>
<script>PF.fn.growl.call("<?php echo (G\get_route_name() == 'user' ? _s('The user has been deleted') : _s('The content has been deleted.')); ?>"); </script>
<?php } ?>

Thats its. You just need a cron which runs your route script for example every hour.
Command: crontab -e

1 * * * *      wget -O /dev/null -o /dev/null https://www.domain.tld/createSitemap

Or use a website like https://www.setcronjob.com/


Todo:

• Private images should not be in a sitemap

 

[Imageforge]

Just what I was hoping for. Thanks for this.

 

[anmolgill]

Warning: DOMDocument::loadXML(): redefinition of the xmlns prefix is forbidden in Entity, line: 2 in /home/app/routes/route.createSitemap.php on line 114

Chevereto error: Internal Server Error

getting error while generating sitemap

 

[evilmoe2]

Its just a warning.

Probably you need to increase your PHP execution time or your memory limit-

 

[Rodolfo]

http://stackoverflow.com/a/17045912

 

[matou19]

I get the error:

This page contains the following errors:
error on line 2 at column 1: Extra content at the end of the document
Below is a rendering of the page up to the first error.

 

[evilmoe2]

Any output before you create the XML?

 

[matou19]

Anythings.

http://lightpics.net/sitemaps/unsorted.xml

 

[evilmoe2]

Do you use a external storage?

I think this query is the problem in your case:

 $q = $DB->query("SELECT image_id, image_title, image_nsfw, DATE_FORMAT(image_date_gmt, '%Y') as year, DATE_FORMAT(image_date_gmt, '%m') as month, DATE_FORMAT(image_date_gmt, '%d') as day, image_storage_mode, image_date_gmt, image_name, image_extension, storage_url FROM chv_images
                          LEFT JOIN chv_storages on storage_id = image_storage_id
                          WHERE image_category_id IS NULL
                          ORDER BY image_date_gmt DESC LIMIT :limit");

 

[matou19]

Yes i use external storage (OVH andGoogle), i have already used your script and its working perfectly, but since recent update in Chevereto, its not working.

 

[evilmoe2]

Any error in log file when calling createSitemap?
What is the content of the xml if you download it with ftp?

 

[matou19]

I have found this error http://pastebin.com/Y0VYyXXt

On the XML File:


<?xml version="1.0"?>

 

[evilmoe2]

I think your have some invalid chars in your title which are included in xml.

$cc_image->addChild("xmlns:image:title", $image["image_title"]);

See here: http://stackoverflow.com/questions/8401222/how-to-validate-cdata-section-for-an-xml-in-php

/**
* Removes invalid characters from an HTML string
*
* @param string $content
*
* @return string
*/
function sanitize_html($content) {
  if (!$content) return '';
  $invalid_characters = '/[^\x9\xa\x20-\xD7FF\xE000-\xFFFD]/';
  return preg_replace($invalid_characters, '', $content);
}

Add this function and replace the line for the title with this.

$cc_image->addChild("xmlns:image:title", APP\sanitize_html($image["image_title"]));

 

[matou19]

Thanks, i have sent you private message. I have error with your code.

 

[SMP]

replace all

$image["image_title"]

with

G\safe_html($image["image_title"])