Kovai Ltd. resolved a lot of bugs in BizTalk360 version 8.8. A whopping 30 bugs were resolved in this version. Let’s talk about one of those bugs which caused some havoc on our system.

Growing tables and not enough disk space

On the 12^th of March, Kovai Ltd. Released version BizTalk360 8.7.1138.0903. The version contained some bug fixes and some great new features.
Unfortunately, some bugs were also introduced in this version.

There was one specific bug, on a specific table, on one specific environment which caused some problems.

After thoroughly testing the version on the UAT environment for one week, we deployed this version during our weekly release window to production. So far so good, no issues during the deployment and for one month, everything seemed fine. Or was I wrong?

7 times 86400 equals a lot!

On Monday the 7^th of May, we started noticing disk performance issues. During the weekend, the disk group – which is monitored by 3^rd party software – failed to notify us that the database disk of our BizTalk cluster was almost full. We still had less than 1 gigabytes of free disk space left.

We identified rather quickly most of the disk space was consumed by the BizTalk360 database. We did a database table check and found one interesting table [dbo].[b360_BizTalkDB_Tbl_sizes].

This table increased with about 7 records each second and that for a steady 48 days before we noticed any issues. During this period, the number of records in the table increased to 29 030 400 million records.

The resolution

We started a few parallel actions to resolve the issue. We quickly asked for more disk storage, we resized some other production databases which were on the same disk and we contacted BizTalk360 support.

They acknowledged the issue and told us it was safe to truncate the table.

Together with the disk extension, this helped as a workaround.

For the next three weeks, we made sure we would truncate the table on time to keep the space in balance.

Version 8.8 is here!

This issue, and about 29 other issues were resolved in version 8.8.

The official release notes:

All the other notes can be found here: https://assist.biztalk360.com/support/solutions/articles/1000264631-release-notes-version-8-8-2116-2405#BF

So, once again, after 1 week of thoroughly testing the version on UAT, we deployed it towards our production systems.

And then…

Then we closed the ticket, we had a large production release, we had some holidays and we kind of forgot the cleanup of the old bug.

Our reports still showed a lot of records, which is by the way normal for our large environment, but the reserved table space was still there.

And this bothered me, since we could use the disk space again. I executed the following command to free up the unused disk space.

Lessons learned

This bug has taught me once again something new or something easily to forget.

No matter how good you test on an acceptance environment, you simply cannot test everything. The acceptance environment is a small environment which is good to test day to day or at most weekly problems. But this issue only caused some troubles after 40+ days, we were not able to see this anytime sooner. Or were we?

Due this issue we also found that the monitoring of the disks could be improved. The disk which contains the BizTalk360 database (and all our other databases) was not monitored correctly.

It only triggered an alert at 1% of free disk space. So, you could say that the issue also resolved another hidden issue.

Due the previous issue, we also realised that it is not a great idea to put your monitoring software on the same hardware which you are monitoring. Thanks to this issue, we also received an additional server to put our monitoring on. Now, both production environments and our monitoring server are separated and carry on doing their own tasks.

BizTalk360 was really on top of this issue and they managed to resolve it quickly before the next release.

On this page you can also find the release notes of the version 8.8 release.