Edit history

I work in this field and can speak to these systems.

These systems tend to perform quite well as their primary selling point is not necessarily cost reduction, flat, but rather increased revenue through higher and more reliable throughput. The trade off is that these systems are quite complex and normally require a great deal of coordination between the customer who may or may not operate the product and the OEM that builds and maintains it.

In my experience the biggest challenge is defining those responsibilities such that throughput is maintained. These systems are like high performance sports cars, they work well if taken care of and operated skillfully but can degrade faster as a result of lack of experience. Some of our customers permit us (the OEM) to operate and maintain to a greater extent then others. Given we have a greater deal of experience with the tool those customers tend to have measurably better performance.

To give an idea of why this particular industry has a harder time building the talent pool I’m referring to consider the domains that need to be covered.

Electromechanical systems, mobile robots, multi-robot systems, PLC programming, DCS, SCADA, multi-axis robots, VFDs, electrical power systems, relational databases, non-relational databases, message brokers, caching systems, distributed systems, Linux and Windows system administration, networking, industrial networking protocols (it’s different), software development, Kubernetes. This is the skill set of one technical professional in this space.

I commonly deal with issues that cover several of those areas simultaneously. E.g. robot communications are slowing down due to full message broker queue which is determined to be caused by a faulty motor causing slippage which leads to a large number of messaged being generated that fill the queue and cause other bots to begin making pathing errors due to communication timeouts.

I work in this field and can speak to these systems.

These systems tend to perform quite well as their primary selling point is not necessarily cost reduction, flat, but rather increased revenue through higher and more reliable throughput. The trade off is that these systems are quite complex and normally require a great deal of coordination between the customer who may or may not operate the product and the OEM that builds and maintains it.

In my experience the biggest challenge is defining those responsibilities such that throughput is maintained. These systems are like high performance sports cars, they work well if taken care of and operated skillfully but can degrade faster as a result of lack of experience. Some of our customers permit us (the OEM) to operate and maintain to a greater extent then others. Given we have a greater deal of experience with the tool those customers tend to have measurably better performance.

To give an idea of why this particular industry has a harder time building the talent pool I’m referring to consider the domains that need to be covered.

Electrical engineering, instrumentation and control equipment, autonomous multi-robot systems, PLC programming, ABB robot programming, Linux system administration, Windows administration, Kubernetes, RabbitMQ, Kafka, Redis, MongoDB, ability to read and modify Python, C# and Rust. These are expected skills for one person to have.

I commonly deal with issues that cover several of those areas simultaneously. E.g. robot communications are slowing down due to full RabbitMQ queue which is determined to be caused by a faulty motor causing slippage which leads to a large number of messaged being generated that fill the queue and cause other bots to begin making pathing errors due to communication timeouts.

EDIT: Removed PII

I work in this field and can speak to these systems.

These systems tend to perform quite well as their primary selling point is not necessarily cost reduction, flat, but rather increased revenue through higher and more reliable throughput. The trade off is that these systems are quite complex and normally require a great deal of coordination between the customer who may or may not operate the product and the OEM that builds and maintains it.

In my experience the biggest challenge is defining those responsibilities such that throughput is maintained. These systems are like high performance sports cars, they work well of taken care of operated skillfully but can degrade faster as a result of lack of experience. Some of our customers permit us (the OEM) to operate and maintain to a greater extent then others. Given we have a greater deal of experience with the tool those customers tend to have measurably better performance.

To give an idea of why this particular industry has a harder time building the talent pool I’m referring to consider the domains that need to be covered.

Electrical engineering, instrumentation and control equipment, autonomous multi-robot systems, PLC programming, ABB robot programming, Linux system administration, Windows administration, Kubernetes, RabbitMQ, Kafka, Redis, MongoDB, ability to read and modify Python, C# and Rust.

I commonly deal with issues that cover several of those areas simultaneously. E.g. robot communications are slowing down due to full RabbitMQ queue which is determined to be caused by a faulty motor causing slippage which leads to a large number of messaged being generated that fill the queue and cause other bots to begin making pathing errors due to communication timeouts.

EDIT: Removed PII

I work for the company that made these robots; at the risk of bias I can speak to this.

The system works great, the problem is Walmart. These systems are MASSIVELY complicated and require quite a bit of technical know-how. Walmart isn’t actually our only customer but they are our biggest. The difference between them and our other customers is Walmart insists on having its locally paid people be the ones to operate and maintain the system (which extends way beyond just the mobile robots and includes more traditional industrial controls and multi-axis robots as well).

Walmart employees are absolute shit at doing this though and Walmart pushes the system way beyond the recommended breaking point without doing the required PMS, this leads to performance reduction over time.

Our contract requires that, prior to a site being brought into production, it has a validation test run where we verify it can maintain the required throughput as a part of the SLA (this throughput is vastly higher than what the warehouse could do without the bots). The sites pass the validation, the issue is throughput degrades over time due to poor training and maintainence practices.

Our other customers don’t maintain the equipment, instead we have paid Symbotic employees at the sites that do it. Those sites have measurably better performance over time and it’s an issue corporate has taken note of.

EDIT: I should add “technical know how” in this case is a rather difficult skillset. I work as a part of reliability and my primary job is RCA for site issues. This job requires you have technical expertise spanning the following areas.

Electrical engineering, instrumentation and control equipment, autonomous multi-robot systems, PLC programming, ABB robot programming, Linux system administration, Windows administration, Kubernetes, RabbitMQ, Kafka, Redis, MongoDB, ability to read and modify Python, C# and Rust.

I commonly deal with issues that cover several of those areas simultaneously. E.g. robot communications are slowing down due to full RabbitMQ queue which is determined to be caused by a faulty motor causing slippage which leads to the queue backing up.

I work for the company that made these robots; at the risk of bias I can speak to this.

The system works great, the problem is Walmart. These systems are MASSIVELY complicated and require quite a bit of technical know-how. Walmart isn’t actually our only customer but they are our biggest. The difference between them and our other customers is Walmart insists on having its locally paid people be the ones to operate and maintain the system (which extends way beyond just the mobile robots and includes more traditional industrial controls and multi-axis robots as well).

Walmart employees are absolute shit at doing this though and Walmart pushes the system way beyond the recommended breaking point without doing the required PMS, this leads to performance reduction over time.

Our contract requires that, prior to a site being brought into production, it has a validation test run where we verify it can maintain the required throughput as a part of the SLA (this throughput is vastly higher than what the warehouse could do without the bots). The sites pass the validation, the issue is throughput degrades over time due to poor training and maintainence practices.

Our other customers don’t maintain the equipment, instead we have paid Symbotic employees at the sites that do it. Those sites have measurably better performance over time and it’s an issue corporate has taken note of.