"But those clouds and servers will not be maintained indefinitely, and once they go down, the cars that depend on them will lose features that owners may be relying upon."
This may prevent cars being advertised for "sale" in California after January 1.
AB 2426: Consumer protection: false advertising: digital goods.[1]
“Digital application or game” means any application or game that a person accesses and manipulates using a specialized electronic gaming device, computer, mobile device, tablet, or other device with a display screen, including any add-ons or additional content for that application or game.
That's a car with an infotainment system.
This law makes it a crime to offer something "for sale" if it can be remotely disabled later, absent a separate acknowledgement that it's a lease.
> This law makes it a crime to offer something "for sale" if it can be remotely disabled later
Solution: stop making vehicles require constant updates and live servers.
The entire point of Android Auto and Apple CarPlay is that you can outsource this task to a device that has a MUCH higher guarantee of support, software quality, and connectivity.
A car infotainment should be able to play local music, control car functions, listen to the radio. None of these require connectivity. You need connectivity for streaming music and GPS. Well phones are really, really good at that.
I mean, that assumes there's anywhere else to go that's doing it differently.
It feels like market forces can push things in either of 2 directions:
(a) People hate practice X, so they shun companies that do it, and enough go to companies who don't do X, which leads to fewer companies doing X
(b) One company gets away with doing X, and despite some complaints, they do just fine – and other companies realize they can get away doing X too, and soon every company is doing it
(c) some companies get away with it because they have a fan base who will lease it anyway. Most companies don't.
People always jump to 0% vs 100%. That almost never happens but rather you get some percentage and that is often enough for a law to be considered successful.
At least it will be honest then. They should also be forced to disclose how they are stealing all your personal info and data as you're driving around. People would think twice before buying any of these cars.
The result? They will not sell any. There is enough old cars around to sustain the transportation needs of the population for quite a while (certainly until someone decides to make new cars without all this crap).
Considering the number of cars I've owned over the decades that couldn't have the entertainment system disabled remotely, I would be fine having those companies with hostile practices exposed.
Same as with cookie banners. Much as I appreciate the intent of the law, any regulation based on disclosure / consent / name-and-shame is bound to fail or make things worse when enshittification is already the standard rather than the exception. There is simply nowhere for customers to take their dollars instead, when manufacturers start rubbing their dark patterns in their customer's faces, as per legal requirement.
Just in case someone from American Lease is reading this, I’d be willing to migrate their servers for less than a million.
Jokes aside, after reading the comments here, I doubt anyone with technical knowledge would believe this. Even with certificate pinning, you can simply dump the firmware as a raw binary, replace the certificate with your own, and upload it back to the car.
And even if the source code is lost, you can still sniff the traffic and implement an API. I did this for my previous employer, who had a collection of expensive, locked devices. It took me about a week, without any prior knowledge or experience. Imagine what someone with more experience could do...
> Even with certificate pinning, you can simply dump the firmware as a raw binary, replace the certificate with your own, and upload it back to the car.
That's assuming they have access to the private key used to sign the firmware though...
Most implementations of this sort of thing in practice don't verify as hard you might think.
A lot of it seems to do with wanting to be able to replace certs and have reasonable expiration times, but not really understanding how to do that (I don't mean it's not possible, i mean the manufacturers seem to not really understand how to do it effectively)
As an example, the siemens CNC controller on my metal mill is totally signed. It has an FPGA with a secure element producing verification signatures to double check cert sigs haven't been modified, Every single file system with binaries is a read-only signed cramfs file signed with a secp521 ecc key. All read-write fsen are mounted noexec, nosuid, etc etc etc.
The initial CA key is baked into secure hardware.
However, in the end, they only verify the CA and signing certs have the right names and properties (various oem specific fields, etc), because the certs have 3-5 year expiration dates and these things are not connected to the internet or even updated often. So they accept expired certs for the signatures, and they also accept any root cert + signing cert that looks the same as the current ones.
So you can replace the CA key and signing keys with something that looks exactly the same as their current one and resign everything, and it works fine.
A whole lot of effort that can be defeated pretty quickly.
I would be surprised if the cars were not similar - they look really secure, but in the end they made tradeoffs that defeat the system.
It would be nice to understand what "can't" means in this context. Are they not technically able to achieve it (which may be overcome by bringing in other technical experts), is it a DNS issue (ha-ha), or is there a licensing issue with some component they integrated with?
I've can't imagine any technical situation short of "we lost the source to the in-car software, which is pinned to a certificate only installed on our server, which only one guy had the password to, and he kept it only in his memory, and he died".
The last place I worked on a system that billed $5-$10 million a month, all pure profit. I was the only one with any knowledge.
When the server was built the guy who did so never documented the root password, and nobody knows it. I was in the sudoers file, and I have no idea what they do since I left. I tried to explain it to multiple levels of management, but I lost them all the second I opened a terminal and ssh’d in.
Ironically, the best solution if that was the case, would be to hire some hackers and hack your own car. Find a kernel exploit, find a power glitching attack against the CPU, find a USB exploit, find a Bluetooth exploit, find a Recovery mode exploit, whatever it may be; and build a physical harness which consistently runs the exploit and modifies the signing certificate. Nasty but effective and probably cheap. Or, even, kills the main GUI and replaces it with your own.
If I was American Lease, I would look into that, besides suing.
Edit: Looking into it, it’s Android Automotive without Google Services. Who wants to bet that it’s an old version of Android on a recycled MediaTek tablet processor which is no longer getting security patches? Knowing MediaTek, completely reprogramming Secure Boot might not be off the table.
Assuming they still have the signing keys, or knowledge of how the build servers worked… or even legal access to the source code, which would require a negotiation / lawsuit against Fisker that could take years. That’s assuming Fisker hasn’t already liquidated that material.
Those are good points, but even hiring someone with knowledge as a consultant could be valuable in framing that case--knowing what you actually need is important.
Another possibility, maybe easier or more sustainable, would be to hire the employees who built the harnesses and interconnects for the car’s entertainment computer. Tell them to build a drop in Raspberry Pi-powered replacement with a custom, basic firmware that only does CarPlay or Android Auto, and call it a day.
Or maybe the result is a Frankenstein creation where the original controller takes care of the driver’s dashboard display, but the entertainment display is Pi-driven.
It's probably way more complicated than that. Ever seen the cold start disaster recovery procedure for a big system with identity and encryption-at-rest and message busses involved? You might be lucky if the bring-up doesnt have any individual stages that take a week to quiesce all by themselves. I know that this system probably isn't all that big, but if I assume their server-side software is as low-quality as their embedded software, I can easily imagine it being that complex and interdependent and poorly documented.
I once saw a small team of FANG engineers, that included two well seasoned senior engineers, revive a project left unmaintained for two years after the owning team was disbanded.
That small team took two weeks alone to get the project to build and run locally, with tons of bits missing.
But hey, if a random anonymous internet expert says that all it takes to revive a project is a week of browsing through the source code then that must be true.
> But hey, if a random anonymous internet expert says that all it takes to revive a project is a week of browsing through the source code then that must be true.
I think the claim of "a week" is probably very wrong, but it's probably possible, at least in some capacity.
However, I think that the actual problem is how badly a lot of software is currently developed. Codebases without proper README files or code comments, even, no proper CI/CD setups in a lot of places and so on. In part, I think it is because developers don't really care about those that will come after them, or because having good discoverability isn't a blocker to get something working or even shipping software. If the situation is absolutely crap in web dev, I fear to think how much worse it is in other industries.
> Sure there is, if you let an outside expert read the source code for a week.
It's mesmerizing how random anonymous people online always have all the answers to the most challenging technical problems conceivable by Man, and they all involve having someone else do the hard/impossible part.
They overestimate tech. I've met some brilliant people, and most of their "secret" was simply their ability to be obsessed with a problem for 12+ hours a day, for days, weeks on end. You can maybe argue they simply process and inerpret information presented faster as well, but that may be a byproduct of the former.
That doesn't necessarily mean they can even layout a plan for something as complex as this for a week, let alone execute anything.
The first thing that came to my mind was "cert pinning and someone lost the private keys", but that seems pretty unlikely in the grand scheme of things that can go wrong.
I'd actually be willing to pick one of these up for a 70%+ discount if it came with all the internal schematics and source code, server migrations be damned.
Likewise, is it strictly "we can't" or "you can't, either" -- the former would be a legal issue whereas the latter might be an tech/IP issue (ownership of PKI and, yes, DNS).
> Fisker informed American Lease that the Oceans "cannot, as a technical matter, be 'ported' from the Fisker server to which the vehicles are currently linked to a distinct server owned and/or controlled by" American Lease.
"as a technical matter" is doing a lot of work in that sentence... I have questions.
I mean, if there's no mechanism for deploying firmware updates, they took off the jtag headers or burned some fuses or blew a programmable-once ROM, there may be no way to deploy updates short of dismantling the car and swapping out a couple major boards. That might well count as "can't".
>The company filed for Chapter 11 bankruptcy protection in June after failing to reach a strategic agreement with another automaker that could provide it with more capital and domestic manufacturing capacity.
I don't think software had anything to do with the filing. they couldn't make and sell cars fast enough and burned through their money over the pandemic.
Are there yet any EVs out there where you actually have control over the software? Or are there any where you could conceivably root the firmware and use open source updates and 3rd party services when the support timeline ends or the manufacturer folds?
My 2019 Hyundai Kona electric is a "dumb" car. Does not even have built-in nav. There is no connectivity, no app, no spying. Hyundai can vanish from existence tomorrow and the car would not even be aware of this happening. I don't care about it's software, because there is nothing to care about.
It's just a dumb-ass car that goes when I need it to go.
It comes with (wired) Android Auto, which is probably it's nicest and most advanced user feature.
My 2019 Kia Niro has some not-worthwhile connected service you can simply say No to. It is then a fully offline car. Assigning ownership involves handing someone the key fob.
I'm thinking that at least with phones you can get an Android and pretty much take it over fully and still have all the services^. I wonder if that will ever happen for cars.
^ Edit: Honestly I don't have any experience with this, but I assume you still need access to Google for notifications etc., so maybe not
Nissan Leaf Visia owner here. The car is the most basic of 4 trim lines. I installed Nissan app and couldn't connect the car and found out that connectivity feature is not available for my basic car.
Analogues of this have been playing out in other industries for some time. The legal incentives strongly bias the outcomes toward it not being possible to own these products, only rent them. At a visceral level I don’t like it but at an intellectual level I understand why it is the only plausible way forward. There is no easy solution.
In automotive, it is common for there to be a requirement that software is supported to some degree for 7-15 years. In practice, this is extremely expensive to guarantee but no one wants to pay for the cost of a reliable guarantee. The industry is at an impasse with consumers and it manifests in situations like this article.
I like renting cars, I hate to take them for some godforsaken garage to fix some issue with it. Sure, there's a premium, and so on, but similarly I hate looking for an overnight parking space, if I can simply hand it over to someone who takes/needs it for the night (going around selling drugs or going to fill it up and clean it, or all of the above) all the better.
with onboard telemetry and ubiquitous HD image recording ability it's pretty easy to make a few shots to have some evidence of good and careful handling, and giving it to the next user in a known condition.
of course if time and space are at a relative abundance then owning might make perfect sense, or if someone uses it so much that handover costs would start to be significant.
7-15 years is nothing for firmware. the hardware is fixed, requirements are also basically fixed. no need to support new codes, new protocols, etc.
A good reminder to never buy a high-dollar tech product that you don't actually own because its continued functioning is dependent upon phoning home to some corporate overlord. I'm happy with my shitty old non-tech-laden car, thanks.
My dream job is solving "impossible" technical problems like this rapidly, on short notice, and eating everything I kill (I'd imagine they could pay $500k+ for a fast [24 hours] solution here).
My company has been doing exactly that for decades. We don't work like you suggest; we charge by the hour and the price is based on the tech, difficulty and urgency. For cases like this it would be €1000/human/hr probably. We also ask intake money. We generally don't work on-site but we can and have been flown in, as far as Australia (we are in the eu); those add a lot of costs. We usually have a handover to the existing or new team, so projects are usually a few weeks to max 3 months (we refuse longer; we specify this in the contract; if it runs over, it's their problem, we are out; it helps getting large companies to cooperate and cut red tape as they tend to find it enough we are there and paying the fees for years if it keeps it all running; I hate that so not doing it).
The reason why my idea is just an idea and yours is a business is yours is a lot more realistic, but I'm happy to hear that something like it exists. I had lost hope that I could find real challenges in my work but now I see that I must look harder.
It is not even hard to find these things, especially if you have some contacts; we definitely cannot handle the work and it's visibly accelerating. There are so many systems where the original team is no longer there and everything 'runs'; especially with VPSs, ec2, docker, kubernetes etc, things are just running blackboxes and when there are changes that don't automatically work and break things, no-one has a clue what to do. A lot of bitrot as well; we had a client who use codeigniter ancient version in 500+ internal and external applications running on ancient php (not supported) etc and with IT demanding updates to the infra, things started failing fast ; a lot of these systems were manually installed on servers that are basically forgotten until something goes down. There are 100s of 1000s of companies in the world that have situations like this and I cannot even attempt to estimate the costs of downtime, hacks, 'must rewrite' new consultants coming in, 'must rewrite' resume driven new employees etc.
I’ve been that guy. It’s fun but it’s not all it’s cracked up to be.
The kinds of people who call you are some of the human beings you would least like to work with, generally. They are almost definitionally over-funded and under-prepared.
Company of a friend of a friend had an intractable problem with an impossible deadline, like, literally, less than a day.
Don’t remember the details, but I fixed it. Deadline next morning, problem solved, saved the day.
It’s midnight, on a status call, wife is sleeping in the next room. The IT guy, his Russian contractor, and “Management” (folks who called me in) are on the call.
The IT guy is SCREAMING at me because I hadn’t checked the code into his convoluted VCS. That thing was a mess. I told them early on that doing that kind of bookkeeping was not a priority. We could deal with it after the crisis had passed.
He was having none of it. At high volume. In the middle of the night.
I simply told them “You know, I don’t have to be here.”
This ][ close to just saying “ok” and hanging up the phone.
This is exactly the same advice as you give an IC that is going after b2b work. It has nothing to do with who they will or won’t work with and everything to do with limiting your liability when playing in the big leagues.
Chances are there is OSS already running in commercially available vehicles.
Owners tinkering with ICE vehicles was and is a thing and I don't see how an electric power train makes that too much different.
Open standards and data formats would be a good middle ground to help avoid the type of problem with Fisker "unable" to migrate to a different provider. Although I wish that vehicles did not have to phone home to the mother ship at all.
The latter one is the most plausible explanation I’ve read yet. Basically, they can’t give away the keys to the kingdom because they’re for the entire kingdom, not just the unsold inventory.
Why would a car even need to have security updates in the first place. Or are we arguing that software updates and patches become like car parts, and you can go to your local parts dealer to get an oil filter, fan belt and a USB with the latest security updates?
In the EU the CRA would have mandated something like that -- not 25 years but a support period defined by the manufacturer that reflects the time the product is expected to be used. The auto industry managed to get themselves excluded.
They might have been licencing third party software that can't be migrated in this way due to contractual limitations.
I imagine the simplest technical solution is for the leasing company to acquire the whole business entity. That may even be what they are trying to do in court.
They're not buying the company though, just a bunch of cars as I understood it. At least several other people have bought Fisker cars as well, perhaps that's where the issue arises.
At the very least, it sounds like a problematic privacy and legal issue for regular Joe to have his car suddenly send data to some "random" third party, without his consent.
Stop stacking the deck with incompetent opportunist scammers who don't belong in front of a computer let alone managing jack shit especially IT projects. Pure stupidity and downvote me all you want.
I’m not overly surprised that in a startup, there was no mechanism built to split off a small defined portion of their fleet to a different, independent cloud.
Bit different to changing the whole infrastructure out for every car in a disaster recovery situation.
I’m also pretty sure they never expected to have to support transferring a car to a different infrastructure. Because if they ever needed to, that meant the company failed. Oh wait…
Short of failure, was buy out and/or IPO with a single purchasor taking it private, or fusion with another company.
There were scenarios down the track requiring porting the customer base into another data model that a competent design team would have in the brief, given a competent leadership.
"But those clouds and servers will not be maintained indefinitely, and once they go down, the cars that depend on them will lose features that owners may be relying upon."
This may prevent cars being advertised for "sale" in California after January 1.
AB 2426: Consumer protection: false advertising: digital goods.[1]
“Digital application or game” means any application or game that a person accesses and manipulates using a specialized electronic gaming device, computer, mobile device, tablet, or other device with a display screen, including any add-ons or additional content for that application or game.
That's a car with an infotainment system.
This law makes it a crime to offer something "for sale" if it can be remotely disabled later, absent a separate acknowledgement that it's a lease.
[1] https://legiscan.com/CA/text/AB2426/2023
> This law makes it a crime to offer something "for sale" if it can be remotely disabled later
Solution: stop making vehicles require constant updates and live servers.
The entire point of Android Auto and Apple CarPlay is that you can outsource this task to a device that has a MUCH higher guarantee of support, software quality, and connectivity.
A car infotainment should be able to play local music, control car functions, listen to the radio. None of these require connectivity. You need connectivity for streaming music and GPS. Well phones are really, really good at that.
Somehow this feels like it could backfire spectacularly.
“Sorry, we can’t ‘sell’ our cars anymore. The law says we have to call them a ‘lease’ and that’ll be be $X,XXX/year”
> “Sorry, we can’t ‘sell’ our cars anymore. The law says we have to call them a ‘lease’ and that’ll be be $X,XXX/year”
Wouldn't that be enough reason to take your business elsewhere?
I mean, that assumes there's anywhere else to go that's doing it differently.
It feels like market forces can push things in either of 2 directions:
(a) People hate practice X, so they shun companies that do it, and enough go to companies who don't do X, which leads to fewer companies doing X
(b) One company gets away with doing X, and despite some complaints, they do just fine – and other companies realize they can get away doing X too, and soon every company is doing it
(c) some companies get away with it because they have a fan base who will lease it anyway. Most companies don't.
People always jump to 0% vs 100%. That almost never happens but rather you get some percentage and that is often enough for a law to be considered successful.
There arent that many console manufacturers / tractor manufacturers / car manufacturers / phone manufacturers / (and unrelated: banks).
Some specialized equipment has few manufacturers.
Arguably this allready happened with computer games.
At least it will be honest then. They should also be forced to disclose how they are stealing all your personal info and data as you're driving around. People would think twice before buying any of these cars.
The result? They will not sell any. There is enough old cars around to sustain the transportation needs of the population for quite a while (certainly until someone decides to make new cars without all this crap).
Considering the number of cars I've owned over the decades that couldn't have the entertainment system disabled remotely, I would be fine having those companies with hostile practices exposed.
That's exactly what we want. Companies to stop selling me shit I can't own.
If they are only willing to lease the car to me, I will happily take my money elsewhere.
Same as with cookie banners. Much as I appreciate the intent of the law, any regulation based on disclosure / consent / name-and-shame is bound to fail or make things worse when enshittification is already the standard rather than the exception. There is simply nowhere for customers to take their dollars instead, when manufacturers start rubbing their dark patterns in their customer's faces, as per legal requirement.
Or, you know, actually design the systems to be handed off at some point. Instead of the Internet Archive we could have the EV Archive.
How does Tesla sell vehicles in California then?
This takes effect 1 JAN 2025.
Just in case someone from American Lease is reading this, I’d be willing to migrate their servers for less than a million.
Jokes aside, after reading the comments here, I doubt anyone with technical knowledge would believe this. Even with certificate pinning, you can simply dump the firmware as a raw binary, replace the certificate with your own, and upload it back to the car.
And even if the source code is lost, you can still sniff the traffic and implement an API. I did this for my previous employer, who had a collection of expensive, locked devices. It took me about a week, without any prior knowledge or experience. Imagine what someone with more experience could do...
> Even with certificate pinning, you can simply dump the firmware as a raw binary, replace the certificate with your own, and upload it back to the car.
That's assuming they have access to the private key used to sign the firmware though...
Most implementations of this sort of thing in practice don't verify as hard you might think.
A lot of it seems to do with wanting to be able to replace certs and have reasonable expiration times, but not really understanding how to do that (I don't mean it's not possible, i mean the manufacturers seem to not really understand how to do it effectively)
As an example, the siemens CNC controller on my metal mill is totally signed. It has an FPGA with a secure element producing verification signatures to double check cert sigs haven't been modified, Every single file system with binaries is a read-only signed cramfs file signed with a secp521 ecc key. All read-write fsen are mounted noexec, nosuid, etc etc etc.
The initial CA key is baked into secure hardware.
However, in the end, they only verify the CA and signing certs have the right names and properties (various oem specific fields, etc), because the certs have 3-5 year expiration dates and these things are not connected to the internet or even updated often. So they accept expired certs for the signatures, and they also accept any root cert + signing cert that looks the same as the current ones.
So you can replace the CA key and signing keys with something that looks exactly the same as their current one and resign everything, and it works fine.
A whole lot of effort that can be defeated pretty quickly.
I would be surprised if the cars were not similar - they look really secure, but in the end they made tradeoffs that defeat the system.
Exactly. This could be as simple as "we stopped paying for the AWS HSM instance and now it's been deleted, the keys are lost"
I suspect this has less to do with the ability to...put the software on another VM and more to do with licensing issues.
[dead]
It would be nice to understand what "can't" means in this context. Are they not technically able to achieve it (which may be overcome by bringing in other technical experts), is it a DNS issue (ha-ha), or is there a licensing issue with some component they integrated with?
I've can't imagine any technical situation short of "we lost the source to the in-car software, which is pinned to a certificate only installed on our server, which only one guy had the password to, and he kept it only in his memory, and he died".
You’d be surprised…
The last place I worked on a system that billed $5-$10 million a month, all pure profit. I was the only one with any knowledge.
When the server was built the guy who did so never documented the root password, and nobody knows it. I was in the sudoers file, and I have no idea what they do since I left. I tried to explain it to multiple levels of management, but I lost them all the second I opened a terminal and ssh’d in.
I can tell it’s still running…
Ironically, the best solution if that was the case, would be to hire some hackers and hack your own car. Find a kernel exploit, find a power glitching attack against the CPU, find a USB exploit, find a Bluetooth exploit, find a Recovery mode exploit, whatever it may be; and build a physical harness which consistently runs the exploit and modifies the signing certificate. Nasty but effective and probably cheap. Or, even, kills the main GUI and replaces it with your own.
If I was American Lease, I would look into that, besides suing.
Edit: Looking into it, it’s Android Automotive without Google Services. Who wants to bet that it’s an old version of Android on a recycled MediaTek tablet processor which is no longer getting security patches? Knowing MediaTek, completely reprogramming Secure Boot might not be off the table.
There's also the option of the ethical version of the $5 wrench hack--hiring the (recently unemployed) people who actually built this stuff.
Assuming they still have the signing keys, or knowledge of how the build servers worked… or even legal access to the source code, which would require a negotiation / lawsuit against Fisker that could take years. That’s assuming Fisker hasn’t already liquidated that material.
Those are good points, but even hiring someone with knowledge as a consultant could be valuable in framing that case--knowing what you actually need is important.
Another possibility, maybe easier or more sustainable, would be to hire the employees who built the harnesses and interconnects for the car’s entertainment computer. Tell them to build a drop in Raspberry Pi-powered replacement with a custom, basic firmware that only does CarPlay or Android Auto, and call it a day.
Or maybe the result is a Frankenstein creation where the original controller takes care of the driver’s dashboard display, but the entertainment display is Pi-driven.
I strongly suspect everyone who knew their infrastructure was laid off (or quit), so there's no chance of a technical handoff
Sure there is, if you let an outside expert read the source code for a week.
It's probably way more complicated than that. Ever seen the cold start disaster recovery procedure for a big system with identity and encryption-at-rest and message busses involved? You might be lucky if the bring-up doesnt have any individual stages that take a week to quiesce all by themselves. I know that this system probably isn't all that big, but if I assume their server-side software is as low-quality as their embedded software, I can easily imagine it being that complex and interdependent and poorly documented.
> It's probably way more complicated than that.
I once saw a small team of FANG engineers, that included two well seasoned senior engineers, revive a project left unmaintained for two years after the owning team was disbanded.
That small team took two weeks alone to get the project to build and run locally, with tons of bits missing.
But hey, if a random anonymous internet expert says that all it takes to revive a project is a week of browsing through the source code then that must be true.
> But hey, if a random anonymous internet expert says that all it takes to revive a project is a week of browsing through the source code then that must be true.
I think the claim of "a week" is probably very wrong, but it's probably possible, at least in some capacity.
However, I think that the actual problem is how badly a lot of software is currently developed. Codebases without proper README files or code comments, even, no proper CI/CD setups in a lot of places and so on. In part, I think it is because developers don't really care about those that will come after them, or because having good discoverability isn't a blocker to get something working or even shipping software. If the situation is absolutely crap in web dev, I fear to think how much worse it is in other industries.
> Sure there is, if you let an outside expert read the source code for a week.
It's mesmerizing how random anonymous people online always have all the answers to the most challenging technical problems conceivable by Man, and they all involve having someone else do the hard/impossible part.
They overestimate tech. I've met some brilliant people, and most of their "secret" was simply their ability to be obsessed with a problem for 12+ hours a day, for days, weeks on end. You can maybe argue they simply process and inerpret information presented faster as well, but that may be a byproduct of the former.
That doesn't necessarily mean they can even layout a plan for something as complex as this for a week, let alone execute anything.
Oh sweet summer child.
I read that as "technically we can't do this as we don't have the money"
The first thing that came to my mind was "cert pinning and someone lost the private keys", but that seems pretty unlikely in the grand scheme of things that can go wrong.
I'd actually be willing to pick one of these up for a 70%+ discount if it came with all the internal schematics and source code, server migrations be damned.
As buggy as their software seems to be, I'd be a little surprised if they worked that hard and actually did it reliably.
Likewise, is it strictly "we can't" or "you can't, either" -- the former would be a legal issue whereas the latter might be an tech/IP issue (ownership of PKI and, yes, DNS).
Yeah...
> Fisker informed American Lease that the Oceans "cannot, as a technical matter, be 'ported' from the Fisker server to which the vehicles are currently linked to a distinct server owned and/or controlled by" American Lease.
"as a technical matter" is doing a lot of work in that sentence... I have questions.
Right, "technically" not theoretically, it is absolutely still possible. This almost always alludes politics.
Yes, 'technically' sounds like bullshit.
It could be something as simple as the server ip address in each car's firmware. That's annoying but not insurmountable.
I mean, if there's no mechanism for deploying firmware updates, they took off the jtag headers or burned some fuses or blew a programmable-once ROM, there may be no way to deploy updates short of dismantling the car and swapping out a couple major boards. That might well count as "can't".
In a way I'm glad fisker went bankrupt if only to provide proof positive of the dangers of the 'connected car'.
>The company filed for Chapter 11 bankruptcy protection in June after failing to reach a strategic agreement with another automaker that could provide it with more capital and domestic manufacturing capacity.
I don't think software had anything to do with the filing. they couldn't make and sell cars fast enough and burned through their money over the pandemic.
I don't believe the person you are replying to was suggesting that software had anything to do with their bankruptcy.
I believe they are suggesting that it is illustrative to the general public as to why buying cars with heavy cloud integration is a bad idea.
This is correct.
Are there yet any EVs out there where you actually have control over the software? Or are there any where you could conceivably root the firmware and use open source updates and 3rd party services when the support timeline ends or the manufacturer folds?
My 2019 Hyundai Kona electric is a "dumb" car. Does not even have built-in nav. There is no connectivity, no app, no spying. Hyundai can vanish from existence tomorrow and the car would not even be aware of this happening. I don't care about it's software, because there is nothing to care about.
It's just a dumb-ass car that goes when I need it to go.
It comes with (wired) Android Auto, which is probably it's nicest and most advanced user feature.
My 2019 Kia Niro has some not-worthwhile connected service you can simply say No to. It is then a fully offline car. Assigning ownership involves handing someone the key fob.
I'm thinking that at least with phones you can get an Android and pretty much take it over fully and still have all the services^. I wonder if that will ever happen for cars.
^ Edit: Honestly I don't have any experience with this, but I assume you still need access to Google for notifications etc., so maybe not
Nissan Leaf Visia owner here. The car is the most basic of 4 trim lines. I installed Nissan app and couldn't connect the car and found out that connectivity feature is not available for my basic car.
A blessing in disguise I guess
I'd guess that making the cars depend on the cloud so much that they are unusable without it is intentional and was a part of the pitch deck.
Analogues of this have been playing out in other industries for some time. The legal incentives strongly bias the outcomes toward it not being possible to own these products, only rent them. At a visceral level I don’t like it but at an intellectual level I understand why it is the only plausible way forward. There is no easy solution.
In automotive, it is common for there to be a requirement that software is supported to some degree for 7-15 years. In practice, this is extremely expensive to guarantee but no one wants to pay for the cost of a reliable guarantee. The industry is at an impasse with consumers and it manifests in situations like this article.
I like renting cars, I hate to take them for some godforsaken garage to fix some issue with it. Sure, there's a premium, and so on, but similarly I hate looking for an overnight parking space, if I can simply hand it over to someone who takes/needs it for the night (going around selling drugs or going to fill it up and clean it, or all of the above) all the better.
with onboard telemetry and ubiquitous HD image recording ability it's pretty easy to make a few shots to have some evidence of good and careful handling, and giving it to the next user in a known condition.
of course if time and space are at a relative abundance then owning might make perfect sense, or if someone uses it so much that handover costs would start to be significant.
7-15 years is nothing for firmware. the hardware is fixed, requirements are also basically fixed. no need to support new codes, new protocols, etc.
A good reminder to never buy a high-dollar tech product that you don't actually own because its continued functioning is dependent upon phoning home to some corporate overlord. I'm happy with my shitty old non-tech-laden car, thanks.
So, just never use any online service. Its a good solution, and so simple!
My dream job is solving "impossible" technical problems like this rapidly, on short notice, and eating everything I kill (I'd imagine they could pay $500k+ for a fast [24 hours] solution here).
Some sort of tech navy seal.
My company has been doing exactly that for decades. We don't work like you suggest; we charge by the hour and the price is based on the tech, difficulty and urgency. For cases like this it would be €1000/human/hr probably. We also ask intake money. We generally don't work on-site but we can and have been flown in, as far as Australia (we are in the eu); those add a lot of costs. We usually have a handover to the existing or new team, so projects are usually a few weeks to max 3 months (we refuse longer; we specify this in the contract; if it runs over, it's their problem, we are out; it helps getting large companies to cooperate and cut red tape as they tend to find it enough we are there and paying the fees for years if it keeps it all running; I hate that so not doing it).
The reason why my idea is just an idea and yours is a business is yours is a lot more realistic, but I'm happy to hear that something like it exists. I had lost hope that I could find real challenges in my work but now I see that I must look harder.
It is not even hard to find these things, especially if you have some contacts; we definitely cannot handle the work and it's visibly accelerating. There are so many systems where the original team is no longer there and everything 'runs'; especially with VPSs, ec2, docker, kubernetes etc, things are just running blackboxes and when there are changes that don't automatically work and break things, no-one has a clue what to do. A lot of bitrot as well; we had a client who use codeigniter ancient version in 500+ internal and external applications running on ancient php (not supported) etc and with IT demanding updates to the infra, things started failing fast ; a lot of these systems were manually installed on servers that are basically forgotten until something goes down. There are 100s of 1000s of companies in the world that have situations like this and I cannot even attempt to estimate the costs of downtime, hacks, 'must rewrite' new consultants coming in, 'must rewrite' resume driven new employees etc.
Sooo ... are you guys looking to hire? I know an DevOps person (me) who's been knee deep in AWS for 10+ years who might be interested. :grin:
I would guess the root of this problem is that they aren't willing to pay.
Indeed. If they did they would just pay their old employees on a contract basis, not this new person who's never seen the code before
I think this must be an engineers wet dream? I'm not a SWE but over the years many engineers have told me that is their dream job.
It's also what all technical consultants do.
I’ve been that guy. It’s fun but it’s not all it’s cracked up to be.
The kinds of people who call you are some of the human beings you would least like to work with, generally. They are almost definitionally over-funded and under-prepared.
That is true; that's why we keep jacking up the hourlies; most of us become more flexible when paid a lot.
Ditto.
Company of a friend of a friend had an intractable problem with an impossible deadline, like, literally, less than a day.
Don’t remember the details, but I fixed it. Deadline next morning, problem solved, saved the day.
It’s midnight, on a status call, wife is sleeping in the next room. The IT guy, his Russian contractor, and “Management” (folks who called me in) are on the call.
The IT guy is SCREAMING at me because I hadn’t checked the code into his convoluted VCS. That thing was a mess. I told them early on that doing that kind of bookkeeping was not a priority. We could deal with it after the crisis had passed.
He was having none of it. At high volume. In the middle of the night.
I simply told them “You know, I don’t have to be here.”
This ][ close to just saying “ok” and hanging up the phone.
Good times.
That's why I have the "eat everything I kill" bit. When I leave they will be much less funded and much better prepared.
That sounds like a fun company idea tbh
That would be pretty fun. Joe Grand has made a career of it, I'm sure there are others.
Yeah form an LLC and pretend there are a bunch of people involved to land that contract
They wont typically pay an individual a rate equivalent to that, but an LLC can get a contract
This is exactly the same advice as you give an IC that is going after b2b work. It has nothing to do with who they will or won’t work with and everything to do with limiting your liability when playing in the big leagues.
Are there any viable open source electric vehicle initiatives out there yet?
As much as I love OSS I wouldn't want it in my literal daily driver.
"My car shuts off when I open all the doors"
"Why would you want it to run? You can't drive with your doors open! Marking as won't fix, working as intended."
"It's open source! You can fix the code yourself!"
No fucking thanks. Some things should cost money and have real stakes.
Chances are there is OSS already running in commercially available vehicles.
Owners tinkering with ICE vehicles was and is a thing and I don't see how an electric power train makes that too much different.
Open standards and data formats would be a good middle ground to help avoid the type of problem with Fisker "unable" to migrate to a different provider. Although I wish that vehicles did not have to phone home to the mother ship at all.
Is it because someone fired hit [<] and [>] buttons on HSM on the way out, or is it because they’re trying to take control of fractions of the fleet?
The latter one is the most plausible explanation I’ve read yet. Basically, they can’t give away the keys to the kingdom because they’re for the entire kingdom, not just the unsold inventory.
Technical incompetence, it sounds like? The risks of technical incompetence seem to be rising.
Probably a bigger problem for modern cars (including EVs) is how long security updates will be provided for.
Why would a car even need to have security updates in the first place. Or are we arguing that software updates and patches become like car parts, and you can go to your local parts dealer to get an oil filter, fan belt and a USB with the latest security updates?
There should be some reasonable mandate how long security updates have to be provided from last manufacturing date. Say 25 years.
In the EU the CRA would have mandated something like that -- not 25 years but a support period defined by the manufacturer that reflects the time the product is expected to be used. The auto industry managed to get themselves excluded.
I guess they're handing over full control of those servers, then. Seems a pretty simple problem to solve, given the legal mandate Fisker is under?
I don’t understand the issue. Just transfer the domain names and the servers to the buyer. Problem solved.
They might have been licencing third party software that can't be migrated in this way due to contractual limitations.
I imagine the simplest technical solution is for the leasing company to acquire the whole business entity. That may even be what they are trying to do in court.
They're not buying the company though, just a bunch of cars as I understood it. At least several other people have bought Fisker cars as well, perhaps that's where the issue arises.
At the very least, it sounds like a problematic privacy and legal issue for regular Joe to have his car suddenly send data to some "random" third party, without his consent.
Stop stacking the deck with incompetent opportunist scammers who don't belong in front of a computer let alone managing jack shit especially IT projects. Pure stupidity and downvote me all you want.
cloud features are the enemy of consumers
Reminder for anyone here who has imposter syndrome:
Highly paid engineers built a system this way.
I also have doubts they were specifically instructed to make the infrastructure non-transferable.
I’m not overly surprised that in a startup, there was no mechanism built to split off a small defined portion of their fleet to a different, independent cloud.
Bit different to changing the whole infrastructure out for every car in a disaster recovery situation.
I’m also pretty sure they never expected to have to support transferring a car to a different infrastructure. Because if they ever needed to, that meant the company failed. Oh wait…
Short of failure, was buy out and/or IPO with a single purchasor taking it private, or fusion with another company.
There were scenarios down the track requiring porting the customer base into another data model that a competent design team would have in the brief, given a competent leadership.
oh wait...
"I'm going to need you to stay late this weekend, we're developing capabilities for a scenario where you've all been laid off."
Are we sure they were highly paid? Maybe that was part of the problem.