We did it! Now indexing 200,000 UPCs! (oh, wait…)

Just to announce a little milestone – since launching back in May-June 2017 we have managed to expand our database of UPCs to 200,000 unique product codes!

However, our counter above tells a different story. Unfortunately we realised we had to cull a large number of invalid codes that had crept into our database. We originally did not bother validating UPCs as retailers have a tendency to be sloppy with the format as well (digits missing etc. etc.) but now we realise that this is actually quite a small percentage and we don’t need that kind of invalid data on our site. However, the size of our UPC database has suffered as a result and is now somewhat below the 200,000 mark again, and it will probably take a week or so now for us to regain our “historic milestone”.

It doesn’t matter – quality is more important than quantity for now. We probably could have got to 200K sooner anyway, but we are trying not to rush things in any case as we tweak the site and its functionality along the way. We also need to see whether we need to invest in greater server capacity in the near future.

Plans for the future include implementing code to fetch search results asynchronously so that page load times are faster (that’s the big one), as well as looking towards gradually superceding the UPC system with GTIN-12/-13 numbers. Yes, GTIN-12s are the same as UPCs, we just need to decide how to reference them, and also how to add EAN-13s into the mix. We haven’t decided how we are going to do that yet in a way that doesn’t disrupt the existing data we have.

Another huge job is to figure out how to internationalise our database so that visitors from, say, the UK, can search products primarily aimed at the UK market, and so on. Currently we focus on US retailers and so a UK searcher may well not find what they are looking for.

So, the next target will be half a million UPCs! We’ll get there first quarter of next year probably (sooner if we are happy with site stability), but we have a lot to do along the way.

Leave a Reply

Your email address will not be published. Required fields are marked *