NPU

[Guest]

hi everybody...  are you planning to migrate the IO scheduler to the NPU?  in most cases, it is not involved on the devices that your operating system serves.  This would give space for new perspectives.

[going]

5 часов назад, Maksimarm сказал:

are you planning to migrate the IO scheduler to the NPU?

@Maksimarm What do you mean by that?

Can you explain it in an expanded form?

[Guest]

I think you understand what an NPU module is!    on some SoCs, it is available but not used.  why not use it to deploy the IO scheduler on it.  the algorithms of the existing Linux kernel schedulers require serious improvements, so building a new algorithm on an unused NPU module would give the Armbian operating system a chance to reach a new level.  You really need this right now.

[Werner]

So this is more or less a feature request. Therefore moved.

[Igor]

2 hours ago, Maksimarm said:

so building a new algorithm on an unused NPU module would give the Armbian operating system a chance to reach a new level.

This is open source world. Once we have it, everyone else will have it to.

 

2 hours ago, Maksimarm said:

You really need this right now.

To maximize our time, emotional and financial loss?  We are already loosing tens of thousands of hours every year for what we provide now. Donations, as only source to cover this wish, supports our current work in around 0.5%, which means yearly budget for 2024 R&D is already long gone. Most of HW dealers and predatory users & competitors not just steal (take) in regular ways, but also generate damages however they can, for fun or to make profits for themselves. Re-thing your question. Before you suggest us what we (anyone that operates in open source) need, we we need something from general public: understanding and funding.

[Guest]

you say yourself that the path you are on now does not give you anything.  your only chance to change everything is startup on your Arbian_board with a unique processor and a unique operating system.      you have the opportunity to take into account all the mistakes in both software and hw and give everyone what they are waiting for.  if it works out, then you will stop being poor))))   if not, then you will live on donations.

[going]

4 часа назад, Maksimarm сказал:

I think you understand what an NPU module is!    on some SoCs, it is available but not used.

The devils are hiding in the details!

 

Some hardware vendors claim that NPUs in their processors work on some operating systems.
This is the OS on their cores.
Which specific chip are you interested in?

 

[going]

33 минуты назад, Maksimarm сказал:

and give everyone what they are waiting for.

And what is the user waiting for?
Can you formulate that?

[Igor]

1 hour ago, Maksimarm said:

that the path you are on now does not give you anything.

Never did. We sponsor your consumerism to the degree we want.

 

1 hour ago, Maksimarm said:

and a unique operating system

It seems you don't understand what open source is. 

 

1 hour ago, Maksimarm said:

and give everyone what they are waiting for

Download what we give you and do whatever you want with that.

 

1 hour ago, Maksimarm said:

if it works out, then you will stop being poor

 

What is the definition of poor? Can poor people invest into software development that you can consume it for free? We are already investing millions into open source, but everyone has spending limits. Since you (public) will never understand yours, we have to stop your nonsense in some way. But if you finance, we can hire, if not, you can hire yourself and do whatever you want.

 

If you succeed, we will download and use your idea. Welcome to open source world.

[Guest]

you ask what people want, I tell you that it is certainly not the Armbian that you give them. you probably don't communicate with ordinary users on other forums. there is a lot of negativity about the Linux kernel.  It's not that we should thank you for giving it this Linux kernel, it's you who should thank us for using it.   we are testing it.  Without us, this kernel would be useless to anyone. only we users make something popular and in demand.  You're not doing this.  we can like any shit and this shit will become a sought-after product.  I hope you understand what I'm talking about. Good luck to you guys.

Edited January 20 by Maksimarm

[Igor]

2 hours ago, Maksimarm said:

I hope you understand what I'm talking about.

It is important that you understand what I am talking about.

 

2 hours ago, Maksimarm said:

about the Linux kernel.

 

We don't represent Linux kernel. Actually nobody does. Linux corporation https://www.linuxfoundation.org/ is the closest entity doing that and even they are considered small and poor in the corporate world they are in.

 

What Armbian (a few people that maintains this place) does? We provide tools and build kernels with small but important changes focused into specific single board computers. We add valuable code that this generic kernel code (https://lwn.net/Articles/915435/) becomes usable. Sometimes only to some degree. And that's it.

 

You want to develop something but you seek sponsors of your idea? No problem, just find someone else. We already burned all yearly budget.

 

2 hours ago, Maksimarm said:

there is a lot of negativity

 

Negativity is shaped by your (over)expectations. From our perspective, they are totally off the chart. Once you get it, you will perhaps even support us. Until then, you will treat us like sh* and we will both be unhappy.

[ag123]

here is some 2 cents comments, if you are meaning NPUs like these:

 

https://github.com/rockchip-linux/rknpu2

 

- hardware interfaces are kept as trade secrets and not published anywhere

 

1st the hardware interfaces are practically undocumented, what they provide is mostly an 'sdk' with some binary blobs

it practically means there is *no way* to use the NPU as those binary blobs in turn depend on device drivers which again are binary blobs (no source)

and there is no hardware documentation any where about the technical details, registers etc. 

if that at least those are published, one could possibly start coding something to test things on the NPU.

 

then that for things like ethos-n78

https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-n78

you can find some info here

https://developer.arm.com/Processors/Ethos-N78

but that it seemed the real SOCs with that chip is no where to be seen let alone any boards found with them.

 

- IO / cpu scheduling 

 

cpu frequency scaling / governors are well documented

https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt

https://docs.kernel.org/scheduler/schedutil.html

https://www.kernel.org/doc/html/latest/scheduler/

and for 'simple' ARM (or RISC V) chips, any sort of 'elaborate' scheduling are probably going to just burn more cpu cycles with little to gain.

but that nevertheless the source codes and the documentations are there so that one can try to develop your own governor if you prefer.

and that the elaborate 'advanced' schedutil governor is already there in the kernel, likely in Armbian as well.

Hence, one can proceed to improve that if one deem that the state-of-the-art currently isn't adequate.

 

and if one wants to do some manual scheduling there is the plain old "nice" command

https://www.scaler.com/topics/linux-nice/

 

----

well my thoughts, scheduling and NPU are 2 unrelated issues, it is possible to handle 'elaborate' scheduling without an NPU, this is currently the state-of-the-art from the 'lowly' ARM boards that we are running, to top tier high core count Intel Xeon / AMD Epyc processors and running all that loads ranging from amazon, google, chatgpt etc, no issues.

 

The other thing being the NPUs itself, currently many SOC IP owners, held their hardware interfaces as *trade secrets* and refused to release them.

You would need to jump that hoop to even use the NPU without any documentation, or else use their proprietary binary blob software, which won't work outside their proprietary binary blob distributions.

 

This withheld *trade secret* about the NPU is the biggest pitfall / trap to those buying those boards with those SOCs and wanting to use the NPU. you get *no support*, *no help*, *no nothing* after you buy the board which purportedly has the NPU. practically *useless*. don't even bother to try it for any 'test' 'AI' stuff, you may at best get a *binary blob demo* and that's it (and it is not anywhere close to even using it for any practical purpose, let alone scheduling).

 

And much more than that, using an NPU practically means that your neural network model must be *quantized*, if you know what that means. All those small NPU hardware normally handles like 8 bit integers, 16 bit integers or at best 16 bit floats. This practically means that you would need to spend a lot of effort to *convert* even ordinary neural networks into the *quantized* form that can be processed by the NPU, if you can't convert that it is  unusable. Even if you managed to convert that there is a risk of lost of precision, e.g. if you convert a 32 bit float down to an 8 bit int, you may practically be quantizing a number space of 4 billion numbers (actually more) to 256 quantized levels. that is the extreme of the information loss, and at the end of the day, if it even works, you may simply get *wrong* results, and again it is practically *useless*.

 

[Guest]

18 hours ago, Igor said:

We don't represent Linux kernel. 

 

but you are using the Linux kernel without improving it with all the flaws that are contained in the Linux kernel. There are a lot of them there.  In general, your position is that of an ostrich with its head in the sand. I wanted to offer you something to change for free, to try to change it, but your reaction shocked me.   We have nothing more to talk about.  Your Armbian does not have a bright future in the Linux kernel.

 

[Guest]

17 hours ago, ag123 said:

And much more than that, using an NPU

 

those who want to do something are looking for ways, those who don't want to are looking for excuses and reasons. you are not moving the world forward, you are pulling it back.

[Gunjan Gupta]

@MaksimarmYou are evading the first ever question asked in reply to your thread, which is what you meant by migrating io scheduler to NPU. As I believe will be the case with most of us, when someone says NPU we will think about Neural Processing Units which are for use with AI inferences and hence something that will be used by end users. It has nothing to do with IO scheduler as explained by ag123 above as well. Hence we did asked you what you mean't by it as it does seem like you are clearly not talking about the same thing or have different understanding of the same.

[Gunjan Gupta]

57 minutes ago, Maksimarm said:

those who want to do something are looking for ways, those who don't want to are looking for excuses and reasons. you are not moving the world forward, you are pulling it back.

Alright, lets move the world forward together. How about you book a session for consultation and pay for the same - https://calendly.com/armbian/consultation?month=2024-01. We can discuss about what needs to be done there. Then you fund the development and we will happily work on it maybe even fulltime depending upon how much you are willing to fund.

[Igor]

2 hours ago, Maksimarm said:

I wanted to offer you something to change for free

 

You are delusional? Offer? From you? You asked a question about the topic you clearly don't understand well, @ag123 sponsored you answer / education / FYI and that's about it.

 

2 hours ago, Maksimarm said:

but you are using the Linux kernel without improving it

 

You clearly have no clue what we do. Do your homework. Insulting us is not going into your favor.

 

2 hours ago, Maksimarm said:

those who want to do something are looking for ways, those who don't want to are looking for excuses and reasons. you are not moving the world forward, you are pulling it back.

 

What are you waiting? If its free for you, invest your time. Ours is not free, so ... good luck.

[ag123]

some additional off-topic blurbs about NPU etc.

https://www.intc.com/news-events/press-releases/detail/1663/intel-accelerates-ai-everywhere-with-launch-of-powerful

Quote

Dec 14, 2023 

NEW YORK--(BUSINESS WIRE)-- At its “AI Everywhere” launch in New York City today, Intel introduced an unmatched portfolio of AI products to enable customers’ AI solutions everywhere — across the data center, cloud, network, edge and PC.

Highlights include:

The Intel® Core™ Ultra mobile processor family, the first built on the Intel 4 process technology and the first to benefit from the company’s largest architectural shift in 40 years, delivers Intel’s most power-efficient client processor and ushers in the age of the AI PC.

The 5th Gen Intel® Xeon® processor family is built with AI acceleration in every core, bringing leaps in AI and overall performance and lowering total cost of ownership (TCO).

Intel CEO Pat Gelsinger showed for the first time an Intel® Gaudi®3 AI accelerator, arriving on schedule next year. (so we are aalready there !)

 

https://arstechnica.com/gadgets/2023/12/amds-new-ryzen-8040-laptop-chips-look-a-lot-like-the-ryzen-7040-cpus/

Quote

AMD's briefing and slide deck spend most of their time talking about Ryzen AI and the dedicated NPU. Though Nvidia dominates the market for server-side AI chips, local NPU hardware from AMD, Intel, and various ARM chipmakers may make it easier to run some generative AI workloads locally. Depending on how those features are implemented, they could be better for privacy since the processing is happening on-device rather than sending data to a server somewhere—when Apple talks about privacy, it often talks about this kind of on-device processing since its chips have included NPUs for years longer than Intel's or AMD's.

https://www.amd.com/en/technologies/xdna.html

 

https://coral.ai/

https://developers.googleblog.com/2022/05/coral-googles-platform-for-edge-ai.html

Quote

Coral, Google’s platform for Edge AI, chooses ASUS as OEM partner for global scale

ASUS IoT already has a long-standing history of collaboration with Coral, being the first partner to release a product using the Coral SoM when they launched the Tinker Edge T development board. ASUS IoT has also integrated Coral accelerators into their enterprise class intelligent edge computers and was the first to release a multi Edge TPU device with the award winning AI Accelerator PCIe Card. Because we have this history of collaboration, we know they share our strong commitment to new innovation in edge computing.

 

https://www.nvidia.com/en-sg/ai-data-science/

Quote

Artificial Intelligence

The Most Advanced AI, Ready for Enterprise

 

https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-n78

Quote

NPU

Ethos-N78

Highly Scalable and Efficient Second-Generation ML Inference Processor

 

https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html

Quote

1. NPU

RK3588 has a NPU(Neural Process Unit) that Neural network acceleration engine with processing performance up to 6 TOPS. Using this NPU module needs to download RKNN SDK which provides programming interfaces for RK3588S/RK3588 chip platforms with NPU. This SDK can help users deploy RKNN models exported by RKNN-Toolkit2 and accelerate the implementation of AI applications

 

https://semiconductor.samsung.com/news-events/news/samsung-electronics-to-strengthen-its-neural-processing-capabilities-for-future-ai-applications/

Quote

Samsung Electronics to Strengthen its Neural Processing Capabilities for Future AI Applications

Samsung Electronics Co., Ltd., a world leader in advanced semiconductor technology, today announced that it will strengthen its neural processing unit (NPU) capabilities to further extend the reach of its artificial intelligence (AI) solutions.

 

https://www.qualcomm.com/news/onq/2023/10/introducing-our-next-gen-intelligent-pc-platforms-the-snapdragon-x-series

Quote

Introducing an all-new naming convention for our next-generation of intelligent PC platforms: The Snapdragon X Series

OCT 11, 2023

Qualcomm Technologies has been pushing the boundaries of innovation in powering consumer and commercial PC experiences since the introduction of the Snapdragon 8cx compute platform. We pioneered connectivity in computing with the world’s first 5G-enabled PC, introduced the world’s first dedicated neural processing unit (NPU) for artificial intelligence (AI) processing in a laptop, and we continue to lead performance per watt, enabling up to multiple days of battery life on a single charge. Today, I’m excited to debut an all-new naming architecture for our platforms to align with how we are revolutionizing the next generation of PC experiences: the Snapdragon X Series.

 

https://www.nxp.com/company/blog/introducing-the-nxp-eiq-neutron-neural-processing-unit-npu:BL-INTRODUCING-THE-NXP-EIQ-NPU

Quote

Introducing the NXP eIQ® Neutron Neural Processing Unit (NPU)

January 4, 2023

 

https://blog.st.com/stm32n6/

Quote

STM32N6: Get a sneak peek at the future of AI-powered MCUs

 April 26, 2023

 

erh, it is getting pretty crowded? and that one can likely be assured none of the offerings are the same in anyway.

if one has a board and wanted to try those NPU stuff, one is most welcomed to go ahead and try and perhaps tell the world what you achieved.

there would be many keen audiences who would like to look at what you did and at least 'wow' about it.

 

And by the way all that 'neural' stuff can mostly run on the *CPU*, and that there are plenty of accelerators for that AVX2, AVX512, SSE2, NEON, SIMD, GPGPU etc and that *GPUs* (Nvidia) has been around and has been the *industrial workhorse* for 'AI' stuff.

 

there is also some thoughts about the NPU cacophony as above, while one start developing say on an NPU, no sooner than one gets started, another 10 competitors released another 20 NPUs and AI accelerators for your to try, and even then while you gets started, your SOC company release a new chip and a *brand new* NPU, obseleting your 'old' NPU while you are just about to 'get started' with you NPU stuff.

 

[Guest]

28 minutes ago, ag123 said:

there is also some thoughts about the NPU cacophony as above, while one start developing say on an NPU, no sooner than one gets started, another 10 competitors released another 20 NPUs and AI accelerators for your to try, and even then while you gets started, your SOC company release a new chip and a *brand new* NPU, obseleting your 'old' NPU while you are just about to 'get started' with you NPU stuff.

 

We don't understand each other. I'm just an ordinary person. But, comrades, judging by the messages above, I think they consider themselves kings of the world only because they were able to mold Armbian out of shit.  but you are right about one thing: in the field of software, you will always be late. The only way is for a strong AI to replace you soon. he will do what you cannot.

[Gunjan Gupta]

3 minutes ago, Maksimarm said:

We don't understand each other. I'm just an ordinary person.

If you are not able to understand what ag123 posted, I am guessing you also have no Idea about what an NPU is. You just took some tech jargons, started a thread to look cool with no Idea about what you are asking for. And when people tried to ask for explanation or tried explaining to you that you are not making sense, then you simply started insulting everyone.

 

If that is the case, I suggest we lock and delete this thread. Its not helping anyone.

[c0rnelius]

7 minutes ago, Gunjan Gupta said:

If that is the case, I suggest we lock and delete this thread. Its not helping anyone.

 

Fun read though.

[Werner]

Player left the game. Case closed.