EDI for Magneto. SAP business one Magento Integration.

There are a number of ways and off-the-shelf solutions to integrate your Magento ecommerce Website to a logistics, ERP or back-office system. The main integration points are:

- - Orders automatically sent from Magento on payment approval and loaded to the back-office
- - Inventory levels synchronisations between magento and back office
- -  Shipping tracking update to Magento and automatic completion of the order

With these three integrations and automatic emails to the customers in place, your Online business is on autopilot. You save hours of keying, eliminate mistakes, eliminate out-of-stock issues and provide the fastest shipping time possible.

The problem with these projects/Solutions in our experience is...

High Cost: 

A quick scan of the market will show you will need to pay $10-20K for the integration software, another $5K to $40K to get it working and if you are using SAP business one or a similar back-office, you are in for new licensing and feature costs.

The consulting blows out as the solution is not readily adaptable to mapping your data and system requirements. We have seen so-called SAP to magento adapters that looked like they were built by a team from the local Kindergarten. Lets face it, Digital/Advertising agencies and programming sweat shops have no idea about workflow and real world order management or supply chain processes.

Poor Reliability:

Many of these solutions use the Magento and back-office web APIs. These interfaces are absolutely dependent on the computer power of both systems. A big batch of magento orders due to a promotion can fail as the back-office system can't process them fast enough.

When something goes wrong they are difficult to troubleshoot and often have no means to re-process steps/orders

The Solution:

The right way to go about an integration project is:

  • Data transfers go into queues and are processed on a first-in first out basis eliminating system overloads, internet issues or bad experence for consumers using the web site
  • All orders, shipments, inventory updates are simple csv files that can be copied or read into a spreadsheet
  • Transfers and workflows are controlled by a simple scheduler ( like cron ) and are easy to read and modify scripts, no obtuse/convoluted object oriented programming requiring a PHD to understand.
  • All data mapping from one system to the other is controlled by "mapping templates" this means its easy to map address line 1 & 2 from magento to a single field for the back office system

So what is the best, simplest, most cost effective way to connect to the back-office systems while still doing the project the right way.

The Answer is:

"EDI". Its been around for years, Its probably already set up for your back office and there are many services on the web that can get your back office EDI ready. Many wholesalers setting up an ecommerce site already have extensive experience in EDI, in this case half the project is done!

All you need is the xPractical Magento EDI adaptor and some mapping and consulting and you are right to go. A few features of our solution are:

  • Super fast magento order extract via direct database access ( we never add records direct to magento via database but its ok to read )
  • Multi store, multi EPR/backend capable
  • No modifications to magento core or order status/state system
  • EDI850, ASN/shipping/856, all the usual and proven standards
  • Full logging of progress in Order History 
  • Email issue notification and integration with the Magento HOLD Order function
  • Simple easy to read template mapping system
  • Works with Magento's bundles, xPractical multi-store inventory module, Configurable product
  • SKU mapping ( because the product IDs in the back-end may be different to the magento product IDs )
  • Advanced ASN processing, can handle multi-ship, part-ship, over ship etc
  • No complex module to install, its one cron job and 4 php files and a file system folder arrangement.
  • Uses FTP and SFTP pick up and drop off to communicate with the EDI service
  • Use magento order filter system to easily identify hold ups, short ships etc
  • Archiving of all incoming and outgoing documents for easy retrieval and re-send
  • All EDI documents are transformed to/from simple CSV file format before read/write to magento, simple to find data mapping errors or bad data.
  • Writes to magento DB are all via standard magento PHP function. Absolutely no use of the very slow Magento import/export functions or the equally troublesome web API system
  • Will not overload your ecommerce store even while processing 1000's of orders or shipment updates in seconds.
  • Extensive checking of data integrity, designed based on our 20 years experience in EDI, B2B ecommerce and industrial automation experience.
  • Works with version 1.8 and fully tested with community 1.9.4

The cost of the solution is $3000 US. You own the code. Its proven in a $XXmillion order per year multi country SAP system environment. You will need consulting, mapping and support services. We can teach your IT team to own it if you wish to go it alone. The whole magento to EDI end of the project with testing will likely be $5000-9000AU

The fastest magento site ever!

Check out swann.com

I have overseen the build of 15+ magento websites and this one is the end result of all the learnings.

My client needed a rebuild of a website, they wanted a result that was clean, not-hacked-up, secure, fast and responsive. They also wanted a solution that was supportable and future proof and not tied to any one supplier ( including me! )

The build team was myself on architecture, magento config, devops, coordination of module suppliers and Dave ( from Swann ) on front-end design, Content management & testing.

Magento is a great and flexible ecommerce solution, however, it has a reputation for:
- Being slow due to its clever EAV database design
- Turning into a mess as developers take short cuts and modify the core files
- Becoming unstable and buggy as modules are added/removed from various suppliers
- Causing owners to be locked into one agency as they are the only ones who can change it.

Here is the approach I took to avoid the pitfalls in Magento builds and deliver the outcome that the customer wanted.

Rule 1.  No mods to the core files!
Rule 2.  Purchased third party modules are never loaded direct via their installers
Rule 3.  Everything under source control

All add-on modules ( about 10 ) were loaded to github. Modman/Composer wrappers were put in place. Composer was then used to build a "stage/test" version of our site. A single composer command brings down a fresh magento core and the latest versions of each of the modules from github. A single bash script then makes it ready to run.

Why? The beauty of composer is its ability to reverse out a module without leaving a mess. There is a black art to magento module selection. We ended up removing well known popular modules as they were not playing nice with other modules. Over the years I have built up a module supplier black/white list, there is a massive difference in the quality of magento modules/add-ons.

Rule 4.  Best practice Infrastructure.

Redis was used for sessions and cache. Mysql and Nginx were tuned for magento. If you are still using php5-fpm you are living in the past. HHVM from Facebook is well and truly production ready. Elasticsearch is a must. GTmetrix and Newrelic are great tools. We didn't even need to look at Varnish ( and its complexities ) to get the speed we wanted.

Rule 5. Bash scripts are your best friend

When ever we need to do dev-ops work we put it in a script. Index rebuild, Cache clear, Deploy stage-to-production. Go-live via webroot folder moves. There are simply too many config files and setups to try to remember. Scripts are easy to document, flexible, easy to read. Unless you are deploying multiple sites I would stay away from complex devops deployment tools

Rule 6. No crap modules. No second chances for modules if support isn't quick.

There are 1000's of modules for magento in the market. We ruthlessly discarded any that caused issue we also we only used module suppliers that provided warranty and good support. Composer allowed us to do this.  Buying $50 modules is false economy, buy the good one and pay for 60 days of support, it will still only cost a few hundred dollars.

Rule 7. Research what the experts are doing.

There are some super Magento techs on the net. Check github and the many forums. You will see the same names again and again. Follow their advice. It will lead to gems like n98-magerun and AOE-scheduler. From their generous open source contributions and articles I have built a best-practice knowledge base that never fails.


The outcome is a site that will pass an audit from tools like magento-mess-detector or by top end agencies that specialise in magento. The faceted search is so fast we were thinking of slowing it down so the user knows its working! The site is light and fast on mobile and the predictive search is very cool. Even the 5 store views for international did not slow it down.

The cost of the build was half that quoted by well know magento agencies. The age of Magento infrastructure as a service in cloud is here and with the right source control approach and module selection you can achieve complex sites without commissioning a single programmer.

Even if you are happy with your existing Magento site I believe by instigating a re-platform project like this you will save thousands per month on hosting and be able to deploy new features and changes at a fraction of the cost compared to the traditional agency approaches. Dave who started this project as a web-marketer with css/html skills now deploys new modules and features without calling in an agency or me. He has the confidence to do this as he knows we have built an environment where its is easy to roll-back changes.

Raspberry PI controller for home/business automation. Easy assemble & no wiring. Under $40us to build

16 relay outputs, 4 protected digital inputs, WIFI, RF network, Raspberry PI engine. 
Repair/assemble in minutes. Real world tough and ready.

In our spare time at Xpractical we like to dive into home automation and the internet of things. This article is not related to our professional services, it's essentially aimed at hobbyists, Engineering teachers and small business with a DIY attitude.

Computing boards/modules are cheap and so are real-world interface modules. The internet of things is a "lego plug and play" world, there is absolutely no reason to try and design/build you own controller from the ground up.

These opto isolated ( very safe ) relay boards are reliable, proven, super cheap and are made by the thousands in china.
They can control 240vac and DC.
Forget about triacs and other clever direct to power interfaces, these relays will likely outlast your project and it it does fail for $7 you can replace it.

The raspberry PI A has more than enough power for your control project. You can program in any language and its easy to connect to the internet. The new PI Zero coming soon is $8!

These RF transceivers are a few dollars each, they are good for 10m through walls or 50m outside. For a few dollars more you can get one with antena & amplifiers to get some more range. We will use them on remote sensors.
( more on that later )

These step down regulator modules are cheaper on ebay than buying the parts! Stay away from trying to design and solder surface mount based projects/modules, leave that to the module suppliers and their proper wave soldering facilities.

The problem with all these "lego blocks" is that your have to wire them together. And if something does fail you have de-soldering to do and the potential to re-wire incorrectly. Also lots of wires looks messy and unprofessional. The solution is to use a PCB layout program like "eagle" and send off to china to get a bunch of "Rats-Nest-Eliminator" (RNE)  circuit boards made up cheaply in batches of 10-20. Here's the one I designed, it's mostly about circuit protection & filtering, mounting, simplification and RNE,  but does have a few CMOS chips ( non surface mount ) to enhance the I/O capability of the PI GPIO.

The board above does the following:

  • Direct connect to the raspberry PI GPIO via underside 26 pin connector.
  • Holds 2 CMOS shift registers to give us 16 latched outputs
  • Direct connect to the relay board via 8 of the CMOS outputs and underside 10 pin connector
  • Have a ribbon cable interface for another expansion relay board
  • Hold a 5 to 3.3v regulator module to power the RF
  • Interface via PI SPI to the RF transceiver module
  • Provide two leds for status and warnings
  • Provide simple input overvoltage/spike protection for 4 digital inputs to the PI
  • Will take 24AC/DC via DC-DC Adjustable LM2596 Step Down Module Buck Converter module ( ebay for $3 ) if not powered by USB.
  • Connects all the main modules together in a space efficient package.

So here is the finished controller. Not one wire needed. All mounting via 4 bolts that act as case spacers. The case is just two sheets of perspex with a few drilled holes.  Can be pulled apart and reassemble in a minute or two. All boards held in place by their connection to the RNE board and via rubber grommets to the base so there is less chance of module damage during temperature changes and when screwing wire into the relays.

I have worked professionally in electrical computer process control in the 80's and 90's and have designed this unit with the lessons learned. Here are the design principles:

  • Good visibility to health/condition, easy to dismantle to clean & fault find.
  • Excellent air circulation. ( electronics mounted outside in cabinets can get very hot )
  • Ugly and simple is fine, no one sees the controller when it's working well.
  • Physical protection from tool slips and drops, sensitive parts should never touch case walls
  • Replacement parts should be cheap and common and held on hand, no wasted time trying to fix modules
  • All parts/modules plug-n-play, should be able to disassemble and reassemble without a manual!
  • Use proven modules, avoid custom complex circuits that can hide future problems.
  • Eliminate solder and 1-3 pin interfaces, 8-16 pin connectors are much more robust and reliable.
  • Simple surge/overvoltage protection. If the parts are inexpensive there is no point going over the top with protection circuits, the unit is just as likely to be destroyed by crushing/dropping.
  • No setting-up buttons, text menus etc etc. No one has time to learn all that. You need a green flashing led saying all is good and a red led flashing a count from 1-10 repeatedly indicating an error. Connect via laptop and WIFI to monitor logs and make programming changes
  • Stay connected periodically to the internet so we can interface to cloud monitoring and time sync services. 
  • Always run watchdog processes over your main control programs

Contact me if you would like to purchase the "Rats-Nest-Eliminator" (RNE) bare board in bulk for project or education or re-seller purposes. 2x Individual bare boards will be $20us ( to cover international postage ) Packs of 20 boards will be $50us.

Cloud integrations - the end of on-premise IT

The number of easy to use cloud applications services has exploded. Dropbox, Jira, Basecamp, Xero, Slack are typical of the modern easy-to-use and feature rich web apps. The issue is that there are so many how do you choose which ones are right for you or your organisation? An assessment criteria I use ( amongst many others ) is to evaluate how well the app integrates with other cloud applications.

Since 2011 there has been a steady flow of cloud application connecter services. https://zapier.com/ and https://ifttt.com/wtf are good examples. Here is a list of more   They range from enterprise IT connector focus to personal productivity focus.

These connector apps all have one thing in common and that is a "recipe" or "workflow" definition comprising a source application, a destination application and some form of data movement between the two. A simple example would be "send me a gmail email when a new contact is added to our zoho CRM service"

In zapier I can see a list of popular apps https://zapier.com/zapbook/apps/#sort=popular&filter=all. This gives me a good idea of which apps are integration friendly. It also on occasions shows me alternative apps to the one I was evaluating.

The maturing of cloud integration and security will be the end of locally hosted business applications such as SAP and Microsoft Dynamics CRM. As you take your first steps into cloud applications, maybe with mature services such as collaborative project management or EDM, don't forget to check out how well the app connects to other apps and services, it may prove to be pivotal to the success or failure of the new cloud application.

Collaboration in google docs

I haven't fired up word or excel in 6 months. Well I did once but that was because somebody sent me an attached document in a email. I'm a convert to google docs and so are two of the clients that I work with.

What really impressed me was that for one of these clients we were developing a response to an RFQ ( Request for quote ), this document was 180 pages long and full of images and tables. We had 4-6 people all working on the document at the same time, it was amazing to see their different coloured cursors moving through the document in real time and seeing big blocks of text and images appearing. The only down side to this was that occasionally the area that you were working on changed dramatically, didn't take to long to get used to this.

Here are the advantages that I see:

- Doc is always saved, no need to hit save, it auto saves after every keystroke
- One version of the truth, no more email attached docs flying around
- Simple to use but with enough features to generate quality business documentation
- The commentary/collaboration features are super easy to use and do the job well

The days of the on-site file server and heavy client side software like office are numbered, even microsoft is moving this way with its office 360 which my corporate friends tell me is working well. We did try to use the Office collaborative web version but it could not handle the big document, I'm told 360 can, but haven't confirmed this.

I still see a place for excel for accounting gurus but for 95% of business users Google has hit the right mark in terms of functionality and ease-of-use. The ease with which it handled the big RFP document sealed the deal for me in stability and performance.

The only hiccup we had in the whole process was that we had to convert the google doc back to word as the RFP stipulated that format!

Give me a call if you want to give up that file server sitting in the corner of your office. With our dual WAN connect to the cloud approach we can give you a business environment that costs less, does more faster and is just as reliable as office on a PC.

The only way to outsource your web project

So you have an idea to build a great new website or app with all sorts of functionality. The only problem is that everyone you talk too tells you it is going to cost $50,000. How about you outsource the build to India, that would be cheaper, wouldn't it?

The answer is yes but in our experience there are some essential management rules you need to follow. Only 2 in 10 IT projects fully achieve the vision and objectives set out by the sponsor. Here is the project environment we need to avoid failure:

1. Quick exits - Be able to exit the outsource relationship quickly with no loss of work/progress, Off-shore developers come and go. If a better opportunity comes up for them you may be stranded mid project.

2. Get a quick read on quality - Don't allow the work to be presented to you at the end of a 6 week code build. It may be a mess and not able to be enhanced or extended. It may perform poorly. Set tasks/objectives that can be done in a day or two at the start of the project.

3. Find the right partners quickly - The differences in understanding requirements and delivering outcomes varies from great to very poor and often there is no correlation between the hourly charge rate and the outsourcer competency. On sites like odesk.com the ratings systems always seem to be 4-5 stars, in reality the difference between the best and the worst is vast.

4. Work in short bursts - Break you project down to small tasks, nothing longer than a few days. Pay as you go for these small tasks. Late in the project if trust/success has been achieved its ok to allow tasks of up to a week.

So how do we achieve these rules. Here are some of the techniques that we put in place:

A. You control the source code repository (SCR). Set yourself up a Github or Bitbucket account. These are cloud based facilities that allow the orderly management of many developers working on the one project. Every change is tracked and can be rolled-back if needed.

B. Use an open source code base. There are excellent free products like magneto ecommerce, Joomla and wordpress blogs, magnolia CMS that can form the base or skeleton of your project

C. Set up cloud based project management environment, Your developers subscribe to this, every task, question, answer is kept in this. All communications are managed here. Nothing gets lost, every instruction and decision is logged and acknowledged

D. Set up your own test and production servers in the Cloud, never rely on the developers equipment. Amazon cloud servers cost only cents per hour

E. Qualify your developers up front, are they familiar with the tools described here

Here is how a typical task would run:
- You define a task in the Cloud Project Management system (CPM)
- Developer is notified and acknowledges understanding in CPM
- Developer books out code from SCR
- Developer sets up copy of project on his/her computers and starts work
- Developer submits a screen cast/video of the outcome to CPM
- You approve, developer submits new code branch to SCR
- Another Senior developer reviews code and allows push from SCR to your test systems if all ok

This allows us to engage 3 or more developers in the early stage of the project to work out who is the best fit for our project.

We are experts in setting up properly managed outsource project environments for clients or individuals wanting to leverage off low costs web development. Our aim is to make the client self sufficient. Once the client has cycled through a number of tasks and sign-offs they can then manage the process themselves. Give us a call if you want to know more

Driving traffic

For small businesses to medium business thinking about going online to sell, driving traffic to the web site to reach a acceptable ROI is the first consideration.

So how do we drive traffic?

Step one is to have a compelling destination, there is no point driving traffic to an online store that no one wants to explore.

To be "compelling" you must have product or service that has at least one of the following attributes:

- Unique and desired
- Cheapest - nobody cares which online store they buy printer cartridges or a mouse from
- Compelling new variant to an existing service/product
- Under a brand with existing healthy awareness
- Proven successful in bricks and mortar and will transition well to online
- Well endorsed by celebs, experts or social
- Provides a user experience that is better than all competition
- Be first to online in your category - hard to do these days

Next step is to think about is "location" to gather  passing traffic. This is just like thinking about a physical stores location to capture foot traffic. In the online world location is all about "linkage"

- Capture an interest group - ie targeted product range, deliver a certain style/look
- Leverage an existing relavent consumer database via EDM
- Leverage existing  successful online assets
- Blow everyone else away in google SEO/SEM
- Get on TV or in magazines via an angle or novelty
- Give good stuff away or deep discount to generate interest ( careful, this is not a sustainable approach )

Now we have sorted location and destination we can think about driving traffic from wider sources, This can be via Affiliates, Paid advertising, Seeding social media, Reviews, Flyers, Member-get-member etc, the list of possibilities is almost endless.

Think about an apple store, great destination and always in a high foot traffic location. They got the basics right and with their super brand awareness only have to gently "drive traffic" using new product announcement advertising