The Japanese term "sensei" isn't so much a title as it is a form of respect for those who have paved the way for you in some way or another. It is often given to teachers, but also to doctors, lawyers, politicians and even members of the church. It can even be used when referring to artists, or basically anyone that has mastered an art form or a specific skill. While its usage tend to be more formal in western dojos, referring only to the head of the school, this is mostly due to the fact it is not a term that is part of our everyday vocabulary.
Usually, addressing a master of another style as "sensei" shouldn't be considered bad manners, since you are showing respect to his accomplishments and the role he is currently playing in your own martial development. However, it is not impossible that some martial artists might take offense at being referred to as what they perceive to be a purely karate title. If this happens, apologize, and mention that, in Japanese, the term is not restricted to karate, but rather used to show respect to teachers, figures of authority and masters of a wide variety of skills, such as medicine or art.
But in the end, if you are going to be working with masters of different styles on a regular basis, you should probably ask them how their regular students refer to them. This will show your willingness to learn from them despite your own different style, as well as your desire to learn part of their martial culture and heritage.