I'm chasing my dreams myself to eventually go into InfoSec. I don't see anyone's opinion should have an influence on yours. Although you want to make sure you understand that business and how it works and that the pay is reasonable enough to accomodate the lifestyle you have or better.
Example I would be happy if all I made was $65K gross each year. The InfoSec average is $85K-$118K! I personally do not care as long as I am in that specialty at this point since it far exceeds what I would gauge my lifestyle at really. At this point it's is what I love to do anyhow plain and simple.
So you need some economics behind that position and dig up some info on that company such as its reputation and things like that. You would be surprised how doing something you love could turn into a burden when you do it too much or because it strictly ties into how much money you make. Once a hobby becomes obligation you could possibly lose interest.
Just double check them and make sure. You don't want a life dream to become a real nightmare. A good example is the high stress related to IT specialties. I want to be in InfoSec but I hear it's a very stressful career. Although the benefits are great and the job growth is extremely promising not including that crazy rate of pay. I need to be willing to except the fact that my love for this specialty is strong enough to do even things I may not want to do.
Not all work is perfect sometimes there are the things you love and the things you'll hate. So it can be a love/hate thing, but I am just pointing out a possibility. I have heard many people wish they were in IT only to hate it after they got in. I set my desired income/lifestyle to live comfortably and accepted many instances having had done some jobs in my past e.g. like the military or cable.
I loved the work but didn't always love every single things I had to do. It's like that in my field so I keep my mind open. I expect to be stressed out sometimes in Information Security I mean you are constantly protecting a companies servers and a breach can mean your job. The field is always changing which means always learning, reading and studying.
So by understanding what to expect about its cons I have to make sure that the pro's are well worth it and that I truly love the work. I feel you need to take the same approach. Remember the grass is not always greener on the other side. Things you should also look into are benefits. Does your current company offer them? Does the new company you want to work for offer them too?
Or will you end up paying out of pocket? So then the question comes in. How much more do you need to make to pay for important things like health insurance and such out of pocket. Those are all things to think about. Anyhow hope that helps good luck!