IAC communication time out with message:degraded sequence number diff

Hello Stefan,

May I know what causes the following message on controller log:

IACInVariable::VerifySequenceNumber First Fault - degraded sequence number diff: 7200

This message occured when timeout on external IAC variable (between controllers).

This IAC consist of only 7 bools (simple data type).

The setting of the IN communication variables is as follow:

Unique ID: 180210101 and is unique on the entired network,

IP address: 172.16.80.180 i.e. source controller's IP

ISP values: mixture of true and false

Attribute: retain hidden,

Interval time: normal,

Expected SIL: same (between safety controller PM865)

Acknowledgement group: auto

Parameter on IAC MMS setting:

Interval time normal: 2000ms

Timeout normal: 6000ms

Application interval time (task time) containing the source variables is 900ms

System version:800xA Base S-FP 5.1.4-2 ver5.10.5632.29860

Thank you so much for your time!


Add New Comment


JD_Ang   

asked 2 months ago
Closed



Best Answer

1

Sorry, no idea but I would guess the HI controller has detected some loss of IAC telegrams.

Ensure task tuning has been applied properly and with ample amounts of Task Offset to allow the controller to crunch some network data in/out before entering next 61131 task execution.

Lack of task offset will push communication ahead, possibly up to the 700ms limit (per 1000ms cycle).

With too much IAC, MMS and other network traffic going on and too little task offset the Ethernet chip will overflow the circular buffer holding received but not ”digested” network telegrams and report an Overrun. Disabling NETBIOS on Control Network computer NICs have helped in many cases.

This drives automatic resending of MMS TCP based telegrams but IAC run over the more lightweight UDP protocol, in which the data is lost and receiver need to wait for next transmission.

Check and compare the Remote System>Controller diagnosis>More>Network Information>Get result with other controllers. There are many counters available. Use Google to translate them, they are not invented by ABB, rather used in most network drivers of computer systems running Ethernet. Some are REALLY bad, eg Late Collision while others are perfectly OK to see, eg up to 5% collisions (put in relation to all sent telegrams) are OK on half duplex but not acceptable on full duplex (PM891).

Later in the network info there is a breakdown of how many telegrams the network filter in the controller has dropped (after flowing through and occupied the relatively small Ethernet ring buffer). Your NETBIOS should show up here; try removing traffic the filter is actively removing from the input queue to the processor. Less undesired traffic leave more room for what is important For the controller.

Using an NE870 as router instead of regular PC with RNRP opens up for a more stringent routing and filtering of what is allowed to enter the Control Network.

Stefan Stromqvist   

answered 2 months ago


 


By JD_Ang on 11/10/2017 | Like (0) | Report

Stefan, thank you so much. Your expertise provides more insight to the issue.


By Stefan Stromqvist on 11/10/2017 | Like (0) | Report

I heard from a colleague that it might happen if you restart one of the ends of the IAC communication link.


By JD_Ang on 11/11/2017 | Like (0) | Report

The timeout happen frequently without restart/donwload. We have many external IACs among many controlers. Majority are fine. A few of them very frequently reports uncertains/warnings and frequently reports timeout in the diagnostic window for communication variable. Some good CVs is having bigger data size than bad CVs.The bad CVs is not exceeding 78 bytes for SIL communication. I am now suspecting time synch issue between controller. Best time quality is TQ6 and sometimes no synch status occur. Also thinking long communication route may cause problem i.e. some CV is passing from one controller to another before reaching destination controller.


By Stefan Stromqvist on 11/11/2017 | Like (0) | Report

Sounds like you have some ideas to continue troubleshooting. I’m no expert, but moving idle time around by changing Task Offset up/down might help if problems are due to lack of ”free time” between IEC61131 task execution. There are diagnostic counters in more recent RNRP Fault Tracer tool that may reveal if routing messages go lost forcing rerouting. Rerouting should not take place too often. Seek help from regional ABB to forward a support case to SE if you want professional assistance with this.


By JD_Ang on 11/11/2017 | Like (0) | Report

Thanks again Stefan. We do have free time of more than 600ms within one scan cycle. Yea it has been raise to local support but I get more information from you than support :)


By Rolf N on 11/17/2017 | Like (0) | Report

Degraded mode appears after download to the server side. Timeout occurs if the set timeout time runs out. Uncertain occurs if the basic 500ms timeout runs out (IAC task timeout set to more than 500ms) Make sure that data in the server end is updated more frequently than the interval in the IAC task used in the client. Check that the execution time in the server task is less than the interval in the IAC client task.

/Rolf


By JD_Ang on 11/17/2017 | Like (0) | Report

Thanks for the comments Rolf. The timeout (12sec) is set at 3 times the slow interval time (4 sec) hardware structure IAC MMS setting. Server application is running at interval 900ms and client application 1000ms. It appear that timeout happens without downloading of server controller. It happens randomly. Is there any limits on maximum IAC variable a controller can send?


Add New Comment


Answers

1

Sorry, no idea but I would guess the HI controller has detected some loss of IAC telegrams.

Ensure task tuning has been applied properly and with ample amounts of Task Offset to allow the controller to crunch some network data in/out before entering next 61131 task execution.

Lack of task offset will push communication ahead, possibly up to the 700ms limit (per 1000ms cycle).

With too much IAC, MMS and other network traffic going on and too little task offset the Ethernet chip will overflow the circular buffer holding received but not ”digested” network telegrams and report an Overrun. Disabling NETBIOS on Control Network computer NICs have helped in many cases.

This drives automatic resending of MMS TCP based telegrams but IAC run over the more lightweight UDP protocol, in which the data is lost and receiver need to wait for next transmission.

Check and compare the Remote System>Controller diagnosis>More>Network Information>Get result with other controllers. There are many counters available. Use Google to translate them, they are not invented by ABB, rather used in most network drivers of computer systems running Ethernet. Some are REALLY bad, eg Late Collision while others are perfectly OK to see, eg up to 5% collisions (put in relation to all sent telegrams) are OK on half duplex but not acceptable on full duplex (PM891).

Later in the network info there is a breakdown of how many telegrams the network filter in the controller has dropped (after flowing through and occupied the relatively small Ethernet ring buffer). Your NETBIOS should show up here; try removing traffic the filter is actively removing from the input queue to the processor. Less undesired traffic leave more room for what is important For the controller.

Using an NE870 as router instead of regular PC with RNRP opens up for a more stringent routing and filtering of what is allowed to enter the Control Network.

Stefan Stromqvist   

answered 2 months ago


 


By JD_Ang on 11/10/2017 | Like (0) | Report

Stefan, thank you so much. Your expertise provides more insight to the issue.


By Stefan Stromqvist on 11/10/2017 | Like (0) | Report

I heard from a colleague that it might happen if you restart one of the ends of the IAC communication link.


By JD_Ang on 11/11/2017 | Like (0) | Report

The timeout happen frequently without restart/donwload. We have many external IACs among many controlers. Majority are fine. A few of them very frequently reports uncertains/warnings and frequently reports timeout in the diagnostic window for communication variable. Some good CVs is having bigger data size than bad CVs.The bad CVs is not exceeding 78 bytes for SIL communication. I am now suspecting time synch issue between controller. Best time quality is TQ6 and sometimes no synch status occur. Also thinking long communication route may cause problem i.e. some CV is passing from one controller to another before reaching destination controller.


By Stefan Stromqvist on 11/11/2017 | Like (0) | Report

Sounds like you have some ideas to continue troubleshooting. I’m no expert, but moving idle time around by changing Task Offset up/down might help if problems are due to lack of ”free time” between IEC61131 task execution. There are diagnostic counters in more recent RNRP Fault Tracer tool that may reveal if routing messages go lost forcing rerouting. Rerouting should not take place too often. Seek help from regional ABB to forward a support case to SE if you want professional assistance with this.


By JD_Ang on 11/11/2017 | Like (0) | Report

Thanks again Stefan. We do have free time of more than 600ms within one scan cycle. Yea it has been raise to local support but I get more information from you than support :)


By Rolf N on 11/17/2017 | Like (0) | Report

Degraded mode appears after download to the server side. Timeout occurs if the set timeout time runs out. Uncertain occurs if the basic 500ms timeout runs out (IAC task timeout set to more than 500ms) Make sure that data in the server end is updated more frequently than the interval in the IAC task used in the client. Check that the execution time in the server task is less than the interval in the IAC client task.

/Rolf


By JD_Ang on 11/17/2017 | Like (0) | Report

Thanks for the comments Rolf. The timeout (12sec) is set at 3 times the slow interval time (4 sec) hardware structure IAC MMS setting. Server application is running at interval 900ms and client application 1000ms. It appear that timeout happens without downloading of server controller. It happens randomly. Is there any limits on maximum IAC variable a controller can send?


Add New Comment


0

Gents, thank you for the comment and help. The root cause is found. It was due to duplicated MAC address in the control network.

From the control hardware manual 3BSE036351:

Reuse of CPU modules replaced from redundant configurations within the
same control network, might cause control network problems due to the MAC
and IP address handling. See MAC and IP Address Handling in Redundant
Configuration on page 49. Such reuse should not be fulfilled unless both the
replaced module and the module previously acting together with it in redundant
configuration are known to be restored from the previous mutual address swap.
It is recommended to set up an IP-config session and use the “Restore factory
settings” option subsequently followed by reassignment of the IP address or
assignment of a new IP address.

And:

in order to avoid network problems while reusing previously used CPU
modules within the same plant:
• The stored swap addresses will be remembered until erased by an IP-config
session (Restore factory settings) or until started up as a backup CPU in new
context (in this case a new swap will take place).
• A CPU running in standalone mode (with RCU terminator fitted) will always
use its own native addresses.

JD_Ang   

answered 5 days ago


 


Add New Comment



Get weekly AKS updates


Partner Exclusive Webinars

 

> – Login to the partner portal to register



Points Redemption Program - Redeem your points for ABB training, Bluetooth speakers and mugs. Terms and conditions >


Earn points when you refer a friend
AKS Referral Program is "Live" - Learn more



AKS Experts


avatar
Ask nikismith   

Rank: 255

I have been a part of the Recording & Control Factory team for 17 years in total, having spent a number of years as a repairs technician withi the manufacturing department, but being in my current role for 9 years now.


avatar
Ask Govindaraj   

Rank: 10

Working in ABB India Operation Center. Have Project engineering and commissioning experience in ABB 800xA, Freelance, AC500.


avatar
Ask Dieter Henkel   

Rank: 22


avatar
Ask Flavio Mussolin   

Rank: 4

ABB AVP, Senior Electronic and Automation Engineer with over 30 years of experience in the field of process automation automotive, pharmaceutical, hollow glass, steel and rolling.


avatar
Ask Rob Lyon   

Rank: 3

info@lymac.co.nz I'm an independent DCS software and commissioning engineer with extensive experience in 800xA and other ABB products. I've worked in almost every corner of the world and still haven't seen it all.


avatar
Ask Harsha.D   

Rank: 7

Tech.Support,software and commisioning engineer with Proficient knowledge in 800xA and its products, RNRP,Networking in general.


avatar
Ask Ronny Lindström   

Rank: 20

ABB Service Engineer


avatar
Ask Sumit Gargav   

Rank: 2

I have worked with Freelance in majority, with 800xA FD and Melody system partly. Also had opportunity to work with Protocols - HART,Profibus,FF & Modbus.


avatar
Ask WvanWees   

Rank: 6

I'm a senior service engineer working for ABB in The Netherlands.


avatar
Ask Stefan Stromqvist   

Rank: 1

I joined ABB in the year of 1994 and has since 1999 been working as a Service & Support Engineer at BU Control Technologies in Västerås, Sweden. My areas of expertise are: 800xA Base, 800xA for Advant Master, Information Management, operating systems, RNRP and Ethernet comms/networking in general.


avatar
Ask MMM   

Rank: 5

ABB PA CT Technical Support


avatar
Ask kstoilov   

Rank: 12

Control System Engineer: 800xA, Compact 800, AC500, AC31-50, Advant Master, Simatic, AC&DC Drives 11 years worked for ABB - Control Systems